{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——LightGBM\n",
    "原始特征+tfidf特征\n",
    "boosting_type参数取值goss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，LightGBM以进行参数调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1</th>\n",
       "      <th>feat_2</th>\n",
       "      <th>feat_3</th>\n",
       "      <th>feat_4</th>\n",
       "      <th>feat_5</th>\n",
       "      <th>feat_6</th>\n",
       "      <th>feat_7</th>\n",
       "      <th>feat_8</th>\n",
       "      <th>feat_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.075886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013158</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013158</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.131579</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.460983</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008244</td>\n",
       "      <td>0.022456</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.124622</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.145988</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 188 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id    feat_1  feat_2  feat_3    feat_4    feat_5  feat_6    feat_7  \\\n",
       "0   1  0.016393     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "1   2  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "2   3  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "3   4  0.016393     0.0     0.0  0.014286  0.315789     0.1  0.131579   \n",
       "4   5  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "\n",
       "     feat_8  feat_9  ...  feat_84_tfidf  feat_85_tfidf  feat_86_tfidf  \\\n",
       "0  0.000000     0.0  ...       0.000000       0.075886       0.000000   \n",
       "1  0.013158     0.0  ...       0.000000       0.000000       0.000000   \n",
       "2  0.013158     0.0  ...       0.000000       0.000000       0.000000   \n",
       "3  0.000000     0.0  ...       0.460983       0.000000       0.008244   \n",
       "4  0.000000     0.0  ...       0.000000       0.124622       0.000000   \n",
       "\n",
       "   feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  feat_90_tfidf  feat_91_tfidf  \\\n",
       "0       0.000000            0.0            0.0       0.000000            0.0   \n",
       "1       0.000000            0.0            0.0       0.000000            0.0   \n",
       "2       0.000000            0.0            0.0       0.000000            0.0   \n",
       "3       0.022456            0.0            0.0       0.000000            0.0   \n",
       "4       0.000000            0.0            0.0       0.145988            0.0   \n",
       "\n",
       "   feat_92_tfidf  feat_93_tfidf  \n",
       "0            0.0            0.0  \n",
       "1            0.0            0.0  \n",
       "2            0.0            0.0  \n",
       "3            0.0            0.0  \n",
       "4            0.0            0.0  \n",
       "\n",
       "[5 rows x 188 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# 这里使用原始特征+tf_idf特征，log(x+1)特征为原始特的单调变换，加上log特征对决策树模型影响不大\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "\n",
    "train1 = pd.read_csv(dpath +\"Otto_FE_train_org.csv\")\n",
    "train2 = pd.read_csv(dpath +\"Otto_FE_train_tfidf.csv\")\n",
    "print(type(train2))\n",
    "#去掉多余的id\n",
    "train2 = train2.drop([\"id\",\"target\"], axis=1)\n",
    "train =  pd.concat([train1, train2], axis = 1, ignore_index=False)\n",
    "del train1\n",
    "del train2\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字，LightGBM不支持字符串格式的特征输入/标签输入\n",
    "y_train = train['target'] #形式为Class_x\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#将类别的形式由Class_x变为0-8之间的整数\n",
    "\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM超参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM的主要的超参包括：\n",
    "1. 树的数目 n_estimators 和 学习率 learning_rate\n",
    "2. 树的最大深度 max_depth 和 树的最大叶子节点数目 num_leaves（注意：XGBoost 只有 max_depth，LightGBM 采用叶子优先的方式生成树，num_leaves很重要，设置成比 2^max_depth 小）\n",
    "3. 叶子结点的最小样本数:min_data_in_leaf(min_data, min_child_samples)\n",
    "4. 每棵树的列采样比例：feature_fraction/colsample_bytree\n",
    "5. 每棵树的行采样比例：bagging_fraction （需同时设置bagging_freq=1）/subsample\n",
    "6. 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)\n",
    "\n",
    "7. 两个非模型复杂度参数，但会影响模型速度和精度。可根据特征取值范围和样本数目修改这两个参数\n",
    "1）特征的最大bin数目max_bin：默认255；\n",
    "2）用来建立直方图的样本数目subsample_for_bin：默认200000。\n",
    "\n",
    "对n_estimators，用LightGBM内嵌的cv函数调优，因为同XGBoost一样，LightGBM学习的过程内嵌了cv，速度极快。\n",
    "其他参数用GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相同的交叉验证分组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#直接调用lightgbm内嵌的交叉验证(cv)，可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证，且速度相对较慢\n",
    "def get_n_estimators(params , X_train , y_train , early_stopping_rounds=10):\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "     \n",
    "    lgbmtrain = lgbm.Dataset(X_train , y_train )\n",
    "     \n",
    "    #num_boost_round为弱分类器数目，下面的代码参数里因为已经设置了early_stopping_rounds\n",
    "    #即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=3,  metrics='multi_logloss' , early_stopping_rounds=early_stopping_rounds,seed=3 )\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:' , cv_result['multi_logloss-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 335\n",
      "best cv score: 0.4893913998754056\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "n_estimators_1 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. num_leaves & max_depth=7\n",
    "num_leaves建议70-80，搜索区间50-80,值越大模型越复杂，越容易过拟合\n",
    "相应的扩大max_depth=7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  6.8min remaining:  3.4min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  9.9min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.7,\n",
       "        importance_type='split', learning_rate=0.1, max_bin=127,\n",
       "        max_depth=7, min_child_samples=20, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=335, n_jobs=4, num_class=9,\n",
       "        num_leaves=31, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'num_leaves': range(50, 90, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10) #50,60,70,80\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48948976226926755\n",
      "{'num_leaves': 60}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAELCAYAAADp1+D/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VeW97/HPNxNjGBPmIQxBRVSULaAVJ8Sh7QFbWw84K4K2UlE8PbX33Htur729r2NPxaFaFRDEAXGobWmtVRxwKAYIyoxAmMMY5nlI+N0/9gpuMZAg2Xvtnfzer9d+JftZz1rr93SX/fVZa2UtmRnOOedcoqSFXYBzzrnaxYPHOedcQnnwOOecSygPHueccwnlweOccy6hPHicc84llAePc865hPLgcc45l1AePM455xIqI+wCklFOTo7l5eWFXYZzzqWU2bNnbzGz3Mr6efBUIC8vj8LCwrDLcM65lCJpdVX6+aE255xzCeXB45xzLqE8eJxzziWUB49zzrmE8uBxzjmXUB48zjnnEsqDxznnXEJ58FSjg6VlTJy+ivU79oddinPOJS0Pnmq0Zc8h/u9bi3jyw6KwS3HOuaTlwVON2japx5DeHXht1lrWbtsXdjnOOZeUPHiq2T2XdSU9TTz+/rKwS3HOuaTkwVPNWjaqy819O/Lm58WsKNkTdjnOOZd0PHji4O5Lu1A3M91nPc45VwEPnjjIaViHWy/MY8rc9SzZuDvscpxzLql48MTJXRd3pmFWBo+9tzTsUpxzLql48MRJk/pZ3HFRJ95esJEF63aGXY5zziUND544GtqvE43rZfLoVJ/1OOdcOQ+eOGpUN5PhF3fm/S8388Wa7WGX45xzSSGuwSPpaklLJBVJevAE/a6TZJIiwfssSRMkzZc0V9KlMX1/I2mtpD3HbGOUpEWS5kl6X1LHmGVlkuYErylxGOpx3XZhHs0bZDHaZz3OOQfEMXgkpQNPAdcA3YEhkrpX0C8bGAnMiGkeBmBmZwEDgEckldf6V6B3Bbv8AoiY2dnAG8BvY5btN7OewWvgqY3s5DSok8FPLu3CJ8u2MHPltkTu2jnnklI8Zzy9gSIzW2Fmh4DJwKAK+v0aeBg4ENPWHfgAwMw2AzuASPC+wMw2HLsRM/vQzMrvU1MAtKuugZyqm/p2pEV2HX737hLMLOxynHMuVPEMnrbA2pj3xUHbUZLOA9qb2VvHrDsXGCgpQ1InoBfQ/iT2PRR4O+Z9XUmFkgokXXsS26kWdTPTueeyrsxcuY1/Fm1N9O6dcy6phHZxQXDobDTwQAWLxxMNqkLgMWA6UFbF7d5EdHb03zHNHc0sAtwAPCapSwXrDQ/CqbCkpOSkxlIVg3u3p03jujwy1Wc9zrnaLZ7Bs46vz1LaBW3lsoEewDRJq4C+wBRJETMrNbP7g3Myg4AmQKVn5yVdAfwHMNDMDpa3m9m64OcKYBpw7rHrmtkYM4uYWSQ3N/fkRloFdTLS+Vn/fL5Ys4MPl2yu9u0751yqiGfwzALyJXWSlAUMBo5eUWZmO80sx8zyzCyP6HmZgWZWKKm+pAYAkgYApWa26EQ7k3Qu8Gywjc0x7U0l1Ql+zwG+A5xwW/Hyo17t6NCsPqOnLvVZj3Ou1opb8JhZKTACeAdYDLxmZgslPSSpsivLWgCfS1oM/AK4uXyBpN9KKgbqSyqW9Ktg0X8DDYHXj7ls+gygUNJc4EPgvyoLsXjJTE9jZP98FqzbxTsLN4VRgnPOhU7+X97fFIlErLCwMC7bLjtiDHj0IzLT0nh7ZD/S0hSX/TjnXKJJmh2cTz8hv3NBgqWnifuv6MaSTbv52/xvXBXunHM1ngdPCL53VmtOb5XNY1OXUlp2JOxynHMuoTx4QpCWJu4f0I0VW/by5znrwy7HOecSyoMnJFd2b8lZbRvz+PtLOeyzHudcLeLBExJJjBrQjbXb9vN6YXHY5TjnXMJ48ITo0tNyOa9DE37/wTIOHK7SjRmccy7lefCESBIPXHkaG3YeYPLMNWGX45xzCeHBE7ILuzSnb+dmPDVtOfsP+azHOVfzefCErHzWU7L7IC8VrA67HOecizsPniRwfl4z+uXn8PRHy9lzsDTscpxzLq48eJLEA1eexra9h5g4fVXYpTjnXFx58CSJnu2bcMUZLXj2o+Xs3H847HKccy5uPHiSyP0DurHrQCnPfboy7FKccy5uPHiSyJltGvPds1ox/tOVbN97KOxynHMuLjx4ksx9V3Rj76FSnv14RdilOOdcXHjwJJluLbMZdE4bJk5fRcnug5Wv4JxzKcaDJwmNvKIbh8qO8MxHy8MuxTnnql1cg0fS1ZKWSCqS9OAJ+l0nySRFgvdZkiZImi9prqRLY/r+RtJaSXuO2UYdSa8G+5ohKS9m2S+D9iWSrqr2gVazTjkN+OG5bXmxYDUbdx4IuxznnKtWcQseSenAU8A1QHdgiKTuFfTLBkYCM2KahwGY2VnAAOARSeW1/hXoXcEuhwLbzawr8CjwcLD97sBg4EzgauAPQW1J7d7++ZgZT31YFHYpzjlXreI54+kNFJnZCjM7BEwGBlXQ79dEQyL2P+27Ax8AmNlmYAcQCd4XmFlFz4weBEwMfn8D6C9JQftkMztoZiuBIioOrqTSvll9ro+0Z/KsNRRv3xd2Oc45V23iGTxtgbUx74uDtqMknQe0N7O3jll3LjBQUoakTkAvoH1V92dmpcBOoHlV6khWIy7viiR+/77PepxzNUdoFxcEh85GAw9UsHg80YAoBB4DpgNxvXWzpOGSCiUVlpSUxHNXVda6cT1u7NOBNz4vZtWWvWGX45xz1SKewbOOr89S2gVt5bKBHsA0SauAvsAUSREzKzWz+82sp5kNApoAS6u6P0kZQGNgaxXqAMDMxphZxMwiubm5JzHM+PrJpV3ITBePv78s7FKcc65axDN4ZgH5kjpJyiJ6gn9K+UIz22lmOWaWZ2Z5QAEw0MwKJdWX1ABA0gCg1MwWVbK/KcCtwe8/Aj4wMwvaBwdXvXUC8oGZ1TjOuGqRXZdbL8zjz3PWsWzT7rDLcc65Uxa34AnOs4wA3gEWA6+Z2UJJD0kaWMnqLYDPJS0GfgHcXL5A0m8lFQP1JRVL+lWw6DmguaQiYBTwYFDHQuA1YBHwD+AeM0upJ67ddXEX6mem89h7PutxzqU+RScFLlYkErHCwsKwy/ia0e8u4YkPivj7vf3o3qZR2OU459w3SJptZpHK+vmdC1LE0H6daVQ3g0ffq+xUl3POJTcPnhTRuF4mw/p1ZuqiTcxduyPscpxz7lvz4Ekht1/Uiab1Mxk91Wc9zrnU5cGTQhrWyeDuS7rw0dISCldtC7sc55z7Vjx4UswtF+SR07AOj7zrsx7nXGry4Ekx9bLSueeyLny2YivTi7aEXY5zzp00D54UNKR3B1o3rssjU5fil8M751KNB08KqpuZzojLuzJ79XY+Wpoc95Vzzrmq8uBJUT/u1Z52Tesx2mc9zrkU48GTorIy0ri3fz7zincyddGmsMtxzrkq8+BJYT88ty2dchoweupSjhzxWY9zLjV48KSwjPQ07rsiny837ubvCyp6KKtzziUfD54U9/2z29CtZUMee28ZZT7rcc6lAA+eFJeeJu6/ohtFm/cwZe43nm/nnHNJx4OnBrjqzFZ0b92Ix95bxuGyI2GX45xzJ+TBUwOkpYkHruzG6q37ePPz4rDLcc65E/LgqSEuP70FPds34Yn3izhYmlIPWHXO1TJxDR5JV0taIqlI0oMn6HedJJMUCd5nSZogab6kuZIujenbK2gvkvSEJAXtr0qaE7xWSZoTtOdJ2h+z7Jl4jjksUnTWs27Hfl6btTbscpxz7rgy4rVhSenAU8AAoBiYJWmKmS06pl82MBKYEdM8DMDMzpLUAnhb0vlmdgR4Olg+A/g7cDXwtpn9a8w2HwF2xmxvuZn1rO4xJpuLuubQO68ZT35YxI8j7ambmR52Sc459w3xnPH0BorMbIWZHQImA4Mq6Pdr4GHgQExbd+ADADPbDOwAIpJaA43MrMCi94l5Abg2dmPBDOh64JVqHk/Sk8SoK7uxaddBXipYHXY5zjlXoXgGT1sg9phPcdB2lKTzgPZm9tYx684FBkrKkNQJ6AW0D9aPPXv+jW0C/YBNZrYspq2TpC8kfSSp37ceUQro27k5F3XN4ZmPlrP3YGnY5Tjn3DeEdnGBpDRgNPBABYvHEw2VQuAxYDpQ1TPmQ/j6bGcD0MHMzgVGAZMkNaqgnuGSCiUVlpSk9h2fR13ZjS17DjHxs1Vhl+Kcc98Qz+BZR3SWUq5d0FYuG+gBTJO0CugLTJEUMbNSM7vfzHqa2SCgCbA0WL/d8bYpKQP4IfBqeZuZHTSzrcHvs4HlQLdjizWzMWYWMbNIbm7uKQw7fOd1aMrlp7dgzMcr2H3gcNjlOOfc18QzeGYB+ZI6ScoCBgNTyhea2U4zyzGzPDPLAwqAgWZWKKm+pAYAkgYApWa2yMw2ALsk9Q3O5dwC/CVmn1cAX5rZ0cNxknKDCx2Q1BnIB1bEcdxJYdSAbuzYd5jxn64KuxTnnPuauAWPmZUCI4B3gMXAa2a2UNJDkgZWsnoL4HNJi4FfADfHLPspMA4oIjp7eTtm2WC+eVHBxcC84PLqN4C7zWzbtxxWyujRtjFXndmScZ+sYMe+Q2GX45xzR8kfIvZNkUjECgsLwy7jlH25cRfXPP4JP720Cz+/6vSwy3HO1XCSZptZpLJ+fueCGuz0Vo34/tltmPDPVWzdczDscpxzDvDgqfHuuyKfA4fLeOaj5WGX4pxzgAdPjdcltyE/OLcdL3y2ms27DlS+gnPOxZkHTy0wsn8+ZUeMpz4sCrsU55zz4KkNOjSvz48j7Xll5lrW7dgfdjnOuVrOg6eW+NnlXQF48oNllfR0zrn48uCpJdo0qceQ3u15vbCYNVv3hV2Oc64W8+CpRe65rCvpaeLx933W45wLT6XBI6mLpDrB75dKuldSk/iX5qpbi0Z1ueWCjvzpi2KKNu8JuxznXC1VlRnPH4EySV2BMURv/DkprlW5uLn7ki7UzUz3WY9zLjRVCZ4jwX3XfgD83sx+DrSOb1kuXpo3rMPt38njb/PW8+XGXWGX45yrhaoSPIclDQFuBf4WtGXGryQXb8P6daZhVgaPTl0adinOuVqoKsFzO3AB8BszWxk8EfTF+Jbl4qlJ/Szu7NeZdxZuYn7xzrDLcc7VMpUGT/AcnHvN7BVJTYFsM3s4AbW5OLrjojya1M9k9NQlYZfinKtlqnJV2zRJjSQ1Az4HxkoaHf/SXDxl183krou78OGSEmav3h52Oc65WqQqh9oam9kuoo+UfsHM+hB90qdLcbde2JGchll+rsc5l1BVCZ4MSa2B6/nq4gJXA9TPyuDuS7rwadEWClZsDbsc51wtUZXgeYjo46uXm9ksSZ0B/yOQGuKmvh1p2agOo99dij+N1jmXCFW5uOB1MzvbzH4SvF9hZtdVZeOSrpa0RFKRpAdP0O86SSYpErzPkjRB0nxJcyVdGtO3V9BeJOkJSQrafyVpnaQ5weu7Mev8Mui/RNJVVam9tqibmc6Iy7oyc9U2Plm2JexynHO1QFUuLmgn6U+SNgevP0pqV4X10oGngGuA7sAQSd0r6JcNjARmxDQPAzCzs4ABwCOSymt9OlieH7yujlnvUTPrGbz+Hmy/OzAYODPo+4egNhe4/vz2tG1Sj0em+qzHORd/VTnUNgGYArQJXn8N2irTGygKZkiHgMnAoAr6/Rp4GIh9PGZ34AMAM9sM7AAiwbmmRmZWYNFvyBeAayupYxAw2cwOmtlKoCiozQXqZKRzb/+uzF27gw++3Bx2Oc65Gq4qwZNrZhPMrDR4PQ/kVmG9tsDamPfFQdtRks4D2pvZW8esOxcYKCkj+IPVXkTvEdc22M7xtjlC0jxJ44O/OapSHUEtwyUVSiosKSmpwvBqlh+e146OzevzyLtLOXLEZz3OufipSvBslXSTpPTgdRNwypdABYfORgMPVLB4PNGAKAQeA6YDZZVs8mmgC9AT2AA8cjL1mNkYM4uYWSQ3tyq5WrNkpqdx3xX5LNqwi3cWbgy7HOdcDZZRhT53AL8HHgWMaAjcVoX11hGdpZRrF7SVywZ6ANOC6wNaAVMkDTSzQuD+8o6SpgNLge3Bdr6xTTPbFNN/LF9d+l1ZHS4w8Jy2PPXhckZPXcqVZ7YiPU1hl+RcrWFmHC4zSo8c4XCpcfjIEUrLjMNlRyg9Ev15uCzaVnrkCIdKoz+/2ccoLTvC4SPG4dIj0e2V2VfrlW/ja33s6LY75tTnl9ecEdexVho8ZrYaGBjbJuk+ojORE5kF5AeHytYRPcF/Q8x2dwI5MducBvybmRVKqg/IzPZKGgCUmtmioN8uSX2JXoxwC9FQRFJrM9sQbO4HwILg9ynApOBuC22IXpAws7Jx10bpaeK+K/IZMekL/jZvPYN6fuOIpHNJKVm+tKN9ostLy6J1lC87up/S6Lpf72OUJeAQtxQ9upGZJjLS08hMF5npaWSki8y06M96WfG/9qoqM56KjKKS4DGzUkkjiP4NUDow3swWSnoIKDSzKSdYvQXwjqQjREPr5phlPwWeB+oBbwcvgN9K6kl0VrYKuCuoY6Gk14BFQClwj5lVdtiu1vpuj9ac3qqIx95bxvfOak1Guj+k1lW/nfsP8/v3l7F176GT+tIuLbPgyz+kL+206Jd1+Zd2RloamRlffWlH35d/sYuGmRlkpEW/3Mu/4DPS0sjKiP7MSC9fFqwbbDsjTWRlpMX0CQLimP1nBn2Phkd62le1pIus9LSj28tMT0uaoxj6NpfPSlprZu0r75maIpGIFRYWhl1GaN5duJHhL87mtz86m+sjNfZjdiHZue8wN4+fwaL1u2jVuG7w5ajjfqlmpKd9o0/sF+1XffS1L9mv+pSvF7P9Cv6LPyMtZj/BDCAzI/m+tJOZpNlmFqms37ed8fhlTzXYgO4tObtdY554fxnX9mxLVobPelz12LnvMDc9N4MvN+7imZt6cUX3lmGX5EJw3G8USbuD8ynHvnYTPVfiaihJjBrQjeLt+3mtcG3lKzhXBTv2HeLG5wpYsnG3h04td9zgMbNsM2tUwSvbzL7tTMmliEu65dKrY1Oe/KCIA4f9lJg7NTv2HeLGcTNYunEPz97ci/5neOjUZn4MxVVIEg9c2Y2Nuw7wysw1YZfjUtj2vYe4YewMlm3ew7O39OKy01uEXZILmQePO64Lu+RwQefmPPXhcvYf8lmPO3nb9h7ihnEzKCrZw5ibe3HZaR46zoPHVeKBK7uxZc9BXvhsVdiluBSzbe8hbhhbwPKSPYy9JcKlHjou4MHjTiiS14xLuuXyzEfL2XOwNOxyXIrYuucgN4wtYOWWvYy7JcIl3Wrfbajc8VXlsQgVXd22NnhUQudEFOnCNWpAN7bvO8yET1eGXYpLAVv3HOTGcTOioXNrhIs9dNwxqjLjeQz4OdE7OrcD/g2YRPQxB+PjV5pLFue0b8KA7i0Z88kKdu47HHY5Lolt2XOQG8ZGQ+e5W8+nX76HjvumqgTPQDN71sx2m9kuMxsDXGVmrwJNK1vZ1QyjBnRj94FSxn26IuxSXJLaEhxeW71tL+NvO5+L8nMqX8nVSlUJnn2SrpeUFryu56uHtvkdDGqJM1o34ntnt2b8pyvZtvdQ2OW4JFOy+yBDxhSwZts+xt96Pt/p6qHjjq8qwXMj0Zt0bg5eNwM3SaoHjIhjbS7J3H9FPvsPl/Hsx8vDLsUlkc27DzBkbAFrt+9j/G3nc6GHjqtEpcETPLr6X8wsJ3j9i5kVmdl+M/s0EUW65NC1RTaDerZl4vRVbN59oPIVXI23edcBhowpYN32/Uy4rTcXdvHQcZWrylVt7YIr2DYHrz9KalfZeq5mGtk/n8NlxtPTfNZT223edYDBYwtYv+MAE24/nwu6NA+7JJciqnKobQLRh6m1CV5/DdpcLZSX04AfndeOlwvWsGHn/rDLcSHZtOsAg8cUsHHnAZ6//Xz6dvbQcVVXleDJNbMJZlYavJ4H/BrJWuxn/btiGE9+UBR2KS4Em4LDaxt3HeD523vTx0PHnaSqBM9WSTdJSg9eNwFb412YS17tmtZn8PkdeK1wLWu37Qu7HJdAG3dGZzqbdh1g4h296d2pWdgluRRUleC5A7ge2AhsAH4E3FaVjUu6WtISSUWSHjxBv+skmaRI8D5L0gRJ8yXNlXRpTN9eQXuRpCckKWj/b0lfSpoXnJNqErTnSdovaU7weqYqtbsTu+eyrkjiifeXhV2KS5ANO/czeMxnbA5C5/w8Dx337VTlqrbVZjbQzHLNrIWZXQtcV9l6ktKBp4BrgO7AEEndK+iXDYwEZsQ0Dwv2fRYwAHhEUnmtTwfL84PX1UH7VKCHmZ0NLAV+GbO95WbWM3jdXVntrnKtGtfl5r4defOLdawo2RN2OS7O1u/Yz+AxBWzZc4gXhvYm4qHjTsG3vUnoqCr06Q0UBZdjHyJ6i51BFfT7NfAwX/1RKkSD6gMAM9sM7AAikloDjcyswMwMeAG4Nuj3rpmV38WygOjtfVwc/eTSLmSlp/G4z3pqtPLQ2brnEBPv6E2vjh467tR82+BRFfq0BWKfm1wctH21Eek8oL2ZvXXMunOBgZIyJHUCegHtg/WLT7TNwB3A2zHvO0n6QtJHkvpVOCBpuKRCSYUlJSVVGJ7LaViHWy/MY8rc9SzdtDvsclwcrAtCZ/ve6EynV0e/S5Y7dd82eE75VjnBobPRwAMVLB5PNFQKid6kdDpQpSeRSfoPoBR4OWjaAHQws3OJztQmSWp07HpmNsbMImYWyc31i/aq6q6LO9MgK4PH3lsadimumhVv38fgMZ8dDZ3zOnjouOqRcbwFknZTccAIqFeFba8jOksp1y5oK5cN9ACmBdcHtAKmSBpoZoXA/TG1TCd63mY7Xz+E9rVtSroN+D7QPzgUh5kdBA4Gv8+WtBzoRjTU3Clq2iCLOy7qxBPvL2Ph+p2c2aZx2CW5arB22z6GjC1g5/7DvHhnH3q2bxJ2Sa4GOe6Mx8yyzaxRBa9sMztuYMWYBeRL6iQpCxhM9A9Ry7e/M7gFT56Z5RE9LzPQzAol1ZfUAEDSAKDUzBaZ2QZgl6S+wdVstwB/CfpdDfx7sI2j1/hKyg0udCB4flA+4LdYrkZDL+pE43qZPDrVZz01wdpt+xg8Jho6Lw310HHVL25PIA1O9I8A3gEWA6+Z2UJJD0kaWMnqLYDPJS0GfkH0xqTlfgqMA4qA5Xx1LudJorOoqcdcNn0xME/SHOAN4G4z23bqI3TlGtfLZPjFnXlv8WbmrN0RdjnuFJSHzu4Dh3n5zj6c46Hj4kDBESkXIxKJWGGhH4k7GXsPltLvtx9yZptGvDi0T9jluG9hzdbo4bU9B0t5aWgfzmrnh03dyZE028wilfWL24zH1S4N6mTwk0u68MmyLcxc6RPKVLNma/RCgj0HS3n5Tg8dF18ePK7a3NS3I7nZdXjk3SX4TDp1rN66l8FjPmPf4TJevrMPPdp66Lj48uBx1aZeVjojLuvKjJXbmL7cb+eXClZt2cvgMQUeOi6hPHhctRrcuz1tGtf1WU8KKA+dA4fLmHRnX78U3iWMB4+rVnUy0hlxeT6fr9nBtCV+B4hktXLLXv51zGccKjvCpGF96d7mG39T7VzcePC4avfjSDs6NKvPI1N91pOMVpTsYfCYzzhcZkwa1oczWnvouMTy4HHVLjM9jXv757Ng3S7eWbgp7HJcjOUlexg8poDSMuOVYX05vZWHjks8Dx4XF9f2bEPn3AY8OnUpR474rCcZFG3ew5AxBZQdMSYN68tprbLDLsnVUh48Li4y0tO474puLNm0m7fmbwi7nFqvaPMehowt4IgZrwz30HHh8uBxcfP9s1pzWstsHn1vKaVlR8Iup9Yq2rybwWMKMINXhvWlW0sPHRcuDx4XN2lp4v4B3VhRspe/zFkfdjm10rJN0dABmDy8D/keOi4JePC4uLrqzJb0aNuIx99fxmGf9STU0k27GTK2AElMHt6Xri08dFxy8OBxcSWJUQO6sWbbPt6YXVz5Cq5aLNm4myFjoqHzyrC+dG3RMOySnDvKg8fF3WWnteDcDk34/fvLOFhapQfJulPw5cZd3DC2gPS08pmOh45LLh48Lu4k8cCA01i/8wCTZ64Nu5wabfGGXdwwdgYZ6dHQ6ZLroeOSjwePS4jvdG1On07NePLDIvYf8llPPCxaH53pZKWnMXn4BXT20HFJyoPHJYQkHrjyNEp2H+SlgtVhl1PjLFq/ixvHFVAnI53Jw/vSKadB2CU5d1xxDR5JV0taIqlI0oMn6HedJJMUCd5nSZogab6kuZIujenbK2gvkvSEJAXtzSRNlbQs+Nk0aFfQr0jSPEnnxXPM7vh6d2pGv/wcnv5oOXsPloZdTo2xcP1ObhhXQN3MaOjkeei4JBe34JGUDjwFXAN0B4ZI6l5Bv2xgJDAjpnkYgJmdBQwAHpFUXuvTwfL84HV10P4g8L6Z5QPvB+8J9l/ed3iwvgvJqAHd2Lb3EM9PXxV2KTXCgnU7uWHsDOp76LgUEs8ZT2+gyMxWmNkhYDIwqIJ+vwYeBg7EtHUHPgAws83ADiAiqTXQyMwKLHrb4xeAa4N1BgETg98nHtP+gkUVAE2C7bgQnNuhKf1Pb8GYj1ew68DhsMtJaQvW7eTGcTNoWCeDycMvoGNzDx2XGuIZPG2B2EuYioO2o4LDXu3N7K1j1p0LDJSUIakT0AtoH6wf+8cgsdtsaWblNwXbCLSsah0use4f0I2d+w/z3Ccrwy4lZc0v3skNYwuC0OlLh+b1wy7JuSoL7eKC4NDZaOCBChaPJxoQhcBjwHSgypdCBbOhk7olsqThkgolFZaU+APM4qlH28Zc06MV4z9dyfa9h8IuJ+XMK97BjeMKyK6byeThfWnfzEPHpZZ4Bs86orOUcu2CtnLZQA9gmqRVQF9giqSImZWa2f1m1tPMBgFNgKXB+u2Os81N5YfQgp+bq1gHAGY2xswiZhbJzc39VgO26+dfAAATnElEQVR2VXf/gG7sOVTKmE9WhF1KSpm7dgc3jZtBo3oeOi51xTN4ZgH5kjpJygIGA1PKF5rZTjPLMbM8M8sDCoCBZlYoqb6kBgCSBgClZrYoOJS2S1Lf4Gq2W4C/BJucAtwa/H7rMe23BFe39QV2xhyScyHp1jKbgee04fl/rmLLnoNhl5MS5qzdwU3Peei41Be34DGzUmAE8A6wGHjNzBZKekjSwEpWbwF8Lmkx8Avg5phlPwXGAUXAcuDtoP2/gAGSlgFXBO8B/g6sCPqPDdZ3SWBk/3wOlpbx9LTlYZeS9L5Ys52bx82gSf1MXr3rAto19dBxqUvR0yEuViQSscLCwrDLqBV+/vpcpsxdz8f/fhktG9UNu5yk9Pma7dz63EyaNsjileF9adukXtglOVchSbPNLFJZP79zgQvVvf3zKTtiPPVhUdilJKXZq7dzy3MzadYwi8keOq6G8OBxoWrfrD7Xn9+eV2auoXj7vrDLSSqzV2/j1vEzaR6EThsPHVdDePC40P3s8q5I4skPfNZTrnDVNm55biY5DbN4dfgFtG7soeNqDg8eF7rWjetxQ+8OvD67mFVb9oZdTuhmrYrOdFo0qsvk4RfQqrGf+3I1iwePSwo/vawLmeniifeXhV1KqGaujIZOy0Z1mTy8r4eOq5E8eFxSaJFdl1svyOPPc9ZRtHl32OWEYsaKrdw2YSatGtflleF9/So/V2N58LikcdclXaiXmc6j79W+WU/Biq3c/vwsWjeuy+RhHjquZvPgcUmjWYMs7rioE2/N28DiDbvCLidhPlu+ldsnREPnleF9aeGh42o4Dx6XVO68qDPZdTMYPXVp2KUkxPTlW7jj+Vm0bVovGjrZHjqu5vPgcUmlcf1MhvXrzNRFm5hXvCPscuJqelE0dNo1rccrwzx0XO3hweOSzu3fyaNp/cwaPev5Z9EW7pg4iw7N6jNpWF9ys+uEXZJzCePB45JOdt1M7rqkC9OWlDB79bawy6l2ny6LznQ6NmvgoeNqJQ8el5RuuaAjOQ3r8Mi7NWvW88myEoZOnEWnnAZMGtaHnIYeOq728eBxSal+VgY/vbQL05dvZfryLWGXUy0+XlrC0ImFdMppwMt39qG5h46rpTx4XNK6oU8HWjWqy+h3l5Lqj+/4aGkJd75QSOec6OE1Dx1Xm3nwuKRVNzOdEZd3pXD1dj5elrqznmlLNjPshUK65DbklWF9adYgK+ySnAuVB49LatdH2tOuaT0eeXdJSs56PvxyM8NfmE3X3IZMurMPTT10nItv8Ei6WtISSUWSHjxBv+skmaRI8D5T0kRJ8yUtlvTLmL4jJS2QtFDSfTHtr0qaE7xWSZoTtOdJ2h+z7Jl4jtlVr6yMNO7tn8+84p28t3hz2OWclA++3MRdL84mv2VDJg3z0HGuXEa8NiwpHXgKGAAUA7MkTTGzRcf0ywZGAjNimn8M1DGzsyTVBxZJegVoCAwDegOHgH9I+puZFZnZv8Zs8xFgZ8z2lptZz+ofpUuEH57blqenLWf01KX0P70FaWkKu6RKvb94Ez956XO6tWrIS0P70KS+h45z5eI54+kNFJnZCjM7BEwGBlXQ79fAw8CBmDYDGkjKAOoRDZldwBnADDPbZ2alwEfAD2M3JknA9cAr1TweF5KM9DRG9s9n8YZdvL1gY9jlVOq9RZu4+6XZnNYqm5eH9vXQce4Y8QyetsDamPfFQdtRks4D2pvZW8es+wawF9gArAF+Z2bbgAVAP0nNg5nQd4H2x6zbD9hkZrG3OO4k6QtJH0nqd6oDc4n3L+e0Ib9FQx59byllR5L3XM/URZv4ycuzOaN1I14a2ofG9TPDLsm5pBPaxQWS0oDRwAMVLO4NlAFtgE7AA5I6m9liorOjd4F/AHOCfrGG8PXZzgagg5mdC4wCJklqVEE9wyUVSiosKSk5tcG5apeeJu4f0I2izXuYMndd2OVU6N2FG/npy7Pp3roRL3roOHdc8QyedXx9NtIuaCuXDfQApklaBfQFpgQXGNwA/MPMDpvZZuCfQATAzJ4zs15mdjGwHTj6p+3BobkfAq+Wt5nZQTPbGvw+G1gOdDu2WDMbY2YRM4vk5uae8uBd9bv6zFZ0b92Ix99bRmnZkbDL+Zp3Fm7kpy9/Tvc2jXlhaB8a1/PQce544hk8s4B8SZ0kZQGDgSnlC81sp5nlmFmemeUBBcBAMyskenjtcgBJDYiG0pfB+xbBzw5EQ2ZSzD6vAL40s+LyBkm5wYUOSOoM5AMr4jNkF09paWLUgG6s2rqPNz9PnlnPPxZs5J6XP6dH28a8OLS3h45zlYhb8AQn/0cA7wCLgdfMbKGkhyQNrGT1p4CGkhYSDbAJZjYvWPZHSYuAvwL3mFnsvfMH882LCi4G5gWXV78B3B2cL3IpqP8ZLTinfRMef38Zh0rDn/W8PX8DIyZ9zlntGvPC0N40quuh41xllIp/lBdvkUjECgsLwy7DHcfHS0u4ZfxMfn1tD27u2zG0Ov4+fwM/e+ULzmnXmIl39CbbQ8fVcpJmm1mksn5+5wKXcvrl53B+XlOe/GAZBw4fe21JYrw1Lxo6Pds38dBx7iR58LiUI4lRA05j066DvDxjTcL3/7d567l38hec66Hj3LfiweNS0gVdmvOdrs15eloR+w6VJmy/f527npGT53BehyY8f0dvGtaJ280/nKuxPHhcyho14DS27DnExOmrE7K/v8xZx8jJX9CrQ1Oev91Dx7lvy4PHpaxeHZty2Wm5PPvxcnYfOBzXff1lzjruf3UOkbxmTLj9fBp46Dj3rXnwuJQ2asBp7Nh3mPGfrorbPv78RTR0zs9rxvMeOs6dMg8el9LOateYK7u3ZNynK9i5r/pnPX/6ophRr82hd6foTKd+loeOc6fKg8elvFFXdmPPwVLGflK9N6T44+xiRr02lz6dmjP+Ng8d56qLB49Leae3asT3zmrN+H+uZOueg9WyzTdmF/Nvb8zlgs4eOs5VNw8eVyPcd0U3Dhwu49mPT33W83rhWn7+xlwu7NKc5249n3pZ6dVQoXOunAePqxG6tmjItee2ZeL0VWzedaDyFY7jtVlr+fc/zuM7XXI8dJyLEw8eV2OM7J9P6RHjD9OWf6v1X521hl+8OY+LuuYw7tYIdTM9dJyLBw8eV2N0bN6A6yPtmDRjDet37D+pdSfPXMMv/jiffvm5jL3FQ8e5ePLgcTXKiMvzAfj9B0VVXmfSjDU8+OZ8LumWy5ibe3noOBdnHjyuRmnbpB5Derfn9cK1rNm6r9L+L89Yzf/403wuPS2XZz10nEsIDx5X49xzWVfS08QTHyw7Yb+XClbzH39awGWn5fLMTR46ziWKB4+rcVo0qsvNfTvy5ufFLC/ZU2GfFwtW8z//vIDLT2/BMz7TcS6h4ho8kq6WtERSkaQHT9DvOkkmKRK8z5Q0UdJ8SYsl/TKm70hJCyQtlHRfTPuvJK2TNCd4fTdm2S+DGpZIuipe43XJ4+5Lu1A3M53H3/vmrOeFz1bxv/68gP6nt+Dpm86jToaHjnOJFLfgkZQOPAVcA3QHhkjqXkG/bGAkMCOm+cdAHTM7C+gF3CUpT1IPYBjQGzgH+L6krjHrPWpmPYPX34PtdwcGA2cCVwN/CGpzNVhOwzrcdmEef523niUbdx9tnzh9Ff/5l4VccUYL/uCh41wo4jnj6Q0UmdkKMzsETAYGVdDv18DDQOxf/RnQQFIGUA84BOwCzgBmmNk+MysFPgJ+WEkdg4DJZnbQzFYCRUFtroYbfnFnGmZl8OjUpQA8/8+V/O8pCxnQvSV/uLGXh45zIYln8LQF1sa8Lw7ajpJ0HtDezN46Zt03gL3ABmAN8Dsz2wYsAPpJai6pPvBdoH3MeiMkzZM0XlLTqtbhaqYm9bMY2q8T/1i4kf/8ywJ+9ddFXNm9JU/dcB5ZGX5607mwhPavT1IaMBp4oILFvYEyoA3QCXhAUmczW0x0dvQu8A9gTtAP4GmgC9CTaGA9cpL1DJdUKKmwpKTkW4zIJaM7LupE43qZvPDZaq46syVPeug4F7p4/gtcx9dnI+2CtnLZQA9gmqRVQF9gSnCBwQ3AP8zssJltBv4JRADM7Dkz62VmFwPbgaVB+yYzKzOzI8BYvjqcVlkdBOuPMbOImUVyc3NPceguWTSqm8n/+8FZ3HlRJw8d55JEPP8VzgLyJXWSlEX0BP+U8oVmttPMcswsz8zygAJgoJkVEj28djmApAZEQ+nL4H2L4GcHoud3JgXvW8fs+wdED8sR7HOwpDqSOgH5wMz4DNklo++d3Zr/+f3uZKZ76DiXDOL2kBEzK5U0AngHSAfGm9lCSQ8BhWY25QSrPwVMkLQQEDDBzOYFy/4oqTlwGLjHzHYE7b+V1JPohQmrgLuCOhZKeg1YBJQG65ThnHMuFDKzsGtIOpFIxAoLC8MuwznnUoqk2WYWqayfH3twzjmXUB48zjnnEsqDxznnXEJ58DjnnEsoDx7nnHMJ5cHjnHMuofxy6gpIKgFWn8ImcoAt1VROmGrKOMDHkqxqylhqyjjg1MbS0cwqvfWLB08cSCqsyrXsya6mjAN8LMmqpoylpowDEjMWP9TmnHMuoTx4nHPOJZQHT3yMCbuAalJTxgE+lmRVU8ZSU8YBCRiLn+NxzjmXUD7jcc45l1AePKdI0ipJ8yXNkVQYtDWTNFXSsuBn08q2kwyOM5ZfSVoXtM2R9N2w66wKSU0kvSHpS0mLJV2Qip/LccaRcp+JpNNi6p0jaZek+1L0MzneWFLxc7lf0kJJCyS9Iqlu8Ay1GZKKJL0aPE+tevfrh9pOTfD01IiZbYlp+y2wzcz+S9KDQFMz+0VYNVbVccbyK2CPmf0urLq+DUkTgU/MbFzwD6c+8D9Isc/lOOO4jxT8TMpJSif6FOA+wD2k2GcS65ix3E4KfS6S2gKfAt3NbH/w3LK/A98F3jSzyZKeAeaa2dPVuW+f8cTHIGBi8PtE4NoQa6l1JDUGLgaeAzCzQ8EDA1PqcznBOFJdf2C5ma0mxT6TCsSOJRVlAPUkZRD9j5oNRJ/+/EawPC6fiQfPqTPgXUmzJQ0P2lqa2Ybg941Ay3BKO2kVjQVghKR5ksanwqEQoBNQQvQptl9IGhc8Qj3VPpfjjQNS7zOJNRh4Jfg91T6TY8WOBVLoczGzdcDvgDVEA2cnMBvYYWalQbdioG1179uD59RdZGbnAdcA90i6OHahRY9lpsrxzIrG8jTQBehJ9P+cj4RYX1VlAOcBT5vZucBe4MHYDinyuRxvHKn4mQAQHC4cCLx+7LIU+UyOqmAsKfW5BME4iOh/4LQBGgBXJ2LfHjynKPivBsxsM/AnoDewSVJrgODn5vAqrLqKxmJmm8yszMyOAGOJji/ZFQPFZjYjeP8G0S/wVPtcKhxHin4m5a4BPjezTcH7VPtMYn1tLCn4uVwBrDSzEjM7DLwJfAdoEhx6A2hH9BxWtfLgOQWSGkjKLv8duBJYAEwBbg263Qr8JZwKq+54Yyn/Ugj8gOj4kpqZbQTWSjotaOoPLCLFPpfjjSMVP5MYQ/j6oamU+kyO8bWxpODnsgboK6m+JPHVv5MPgR8FfeLymfhVbadAUmeiMwOIHhaZZGa/kdQceA3oQPQu19eb2baQyqySE4zlRaKHDgxYBdwVc0w+aUnqCYwDsoAVRK84SiP1PpeKxvEEqfmZNCD6ZdfZzHYGbSn3bwWOO5aU+7ci6f8A/wqUAl8AdxI9pzMZaBa03WRmB6t1vx48zjnnEskPtTnnnEsoDx7nnHMJ5cHjnHMuoTx4nHPOJZQHj3POuYTy4HHOOZdQHjzOpQhJz0v6UeU9nUtuHjzOOecSyoPHuVMgKS94QNvY4IFa70qqJ2mapEjQJyd41hGSbpP05+ChZ6skjZA0Krj7dIGkZlXcby9JHwV3En8n5n5nwyTNkjRX0h+D26E0lrRaUlrQp4GktZIyJXWR9I9gO59IOj3o82NFHw42V9LHcfkfz9VaHjzOnbp84CkzOxPYAVxXSf8ewA+B84HfAPuCu09/BtxS2c4kZQK/B35kZr2A8cF2IPoAr/PN7BxgMTA0uKXLHOCSoM/3gXeCG0OOAX4WbOffgD8Eff4TuCrYzsDKanLuZGRU3sU5V4mVZjYn+H02kFdJ/w/NbDewW9JO4K9B+3zg7Crs7zSi4TU1em9H0onehh+gh6T/CzQBGgLvBO2vEr0n14dEnyHzB0kNgQuB14PtANQJfv4TeD54KuWbVajJuSrz4HHu1MXeQLEMqEf0povlRxTqnqD/kZj3R6jav0kBC83sggqWPQ9ca2ZzJd0GXBq0TwH+X3AorxfwAdHnr+wws57HbsTM7pbUB/geMFtSLzPbWoXanKuUH2pzLj5WEf2Ch69uMV9dlgC5ki6A6KE3SWcGy7KBDcHhuBvLVzCzPcAs4HHgb8FzY3YBKyX9ONiOJJ0T/N7FzGaY2X8SfQpq+2oeg6vFPHici4/fAT+R9AWQU50bNrNDRMPsYUlziZ6/uTBY/L+AGUQPlX15zKqvAjcFP8vdCAwNtrOQ6BMpAf5b0nxJC4DpwNzqHIOr3fyxCM455xLKZzzOOecSyi8ucC7JSHoK+M4xzY+b2YQw6nGuuvmhNueccwnlh9qcc84llAePc865hPLgcc45l1AePM455xLKg8c551xC/X8n3wSQ156yEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_means, yerr=test_stds,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_means, yerr=train_stds,label = ' Train')\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'Log Loss' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.49137832, -0.48948976, -0.4904535 , -0.49047327])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 性能抖动，取系统推荐值：70, 不必再细调"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. min_child_samples\n",
    "叶子节点的最小样本数目\n",
    "\n",
    "叶子节点数目：70，共9类，平均每类8个叶子节点\n",
    "每棵树的样本数目数目最少的类（稀有事件）的样本数目：200 * 2/3 * 0.7 = 100\n",
    "所以每个叶子节点约100/8 = 12个样本点\n",
    "\n",
    "搜索范围：10-50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  6.8min remaining:  3.4min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  9.8min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.7,\n",
       "        importance_type='split', learning_rate=0.1, max_bin=127,\n",
       "        max_depth=7, min_child_samples=20, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=335, n_jobs=4, num_class=9,\n",
       "        num_leaves=70, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'min_child_samples': range(10, 50, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48171068722\n",
      "{'min_child_samples': 10}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VeW59/HvTZjDPEPCDKLMQwy0\nUg9SBywKiloVpVQp+rba1qkO1Q7qqVqPx6Ee2lNABquIvoCAHtE6oVZfAkmYwjxDwhDGMGa+3z/2\nxpPSkIQhWXsnv8915UrW2s/a+36ykv3b61lrP9vcHRERkdOpFnQBIiIS2RQUIiJSIgWFiIiUSEEh\nIiIlUlCIiEiJFBQiIlIiBYWIiJRIQSEiIiVSUIiISImqB13A+dCsWTPv0KFD0GWIiESVlJSUfe7e\nvLR2lSIoOnToQHJyctBliIhEFTPbVpZ2GnoSEZESKShERKRECgoRESmRgkJEREqkoBARkRIpKERE\npEQKChERKZGCQkQkCu0/msOT763icHZeuT9WpXjDnYhIVVFY6Ly1ZDvPf7iOYzn5XNK5GZd3b1mu\nj6mgEBGJEmkZWTw+N43lOw4xsGMT/v26nnRtWb/cH1dBISIS4bJO5PHi39fxt0XbaBJbi5dv7svI\nvm0wswp5fAWFiEiEcnfmLsvgD/+zlgPHchgzqD0PXNmNhnVqVGgdCgoRkQi0Yc8RnpibRtKWA/Rp\n24hpd1xMz7iGgdSioBARiSDHc/P506cbmfzVZmJrVeeZ63txy8VtqVatYoaZiqOgEBGJAO7OR6v2\n8NR7q9iZlc1NA+J59OoLaVqvVtClKShERIK2ff9xfjc/jc/X7eXCVvX50639SOjQJOiyvqWgEBEJ\nSHZeARO/3MyEzzdSvZrxxPCLGPvdDtSIiaz3QisoREQC8OX6vfx2Xhpb9x9neO/W/GZ4d1o1rB10\nWcVSUIiIVKBdWSf49/fX8D8rd9GxWSx/G5fI97qW+rHVgVJQiIhUgLyCQqZ/s5WXPl5PfqHz4BUX\ncNe/daJW9ZigSyuVgkJEpJwt2XqAJ95NY92eIwy9sAW/v7YH7ZrWDbqsMlNQiIiUk31Hc3huwVpm\npaQT16gOE8cM4IruLSts6o3zpUyn1s1smJmtM7ONZvZoCe1uNDM3s4Twcg0zm25mK81sjZk9dkr7\nGDNbambvF1nX0cySzGyDmb1tZjXPtnMiIkEoKHTeWLSNoS8sZO7SDH46pDMfP3ApV/ZoFXUhAWU4\nojCzGGACcAWQDiwxs/nuvvqUdvWBXwBJRVbfBNRy915mVhdYbWZvufvW8O2/BNYADYps80fgJXef\naWb/DYwD/nJWvRMRqWAr07N4Yu5KlqdnMahTE54eWTEzvJanshxRJAIb3X2zu+cCM4GRxbR7Gnge\nyC6yzoFYM6sO1AFygcMAZhYPDAcmn2xsoagdCswKr5oOXHcmHRIRCULWiTx+Oy+NERP+QcahbF65\npS9vjR8U9SEBZTtHEQfsKLKcDgws2sDM+gFt3f19M3uoyE2zCIXKLqAucL+7Hwjf9jLwMFD0t9gU\nOOTu+UUeK66MfRERqXDuzrtLM3jmgzUcOJbL2O904IErL6BB7Yqd4bU8lSUoihtQ829vNKsGvAT8\nuJh2iUAB0AZoDHxlZp8A3YFMd08xsyFlfax/KsrsLuAugHbt2pXaCRGR8239niP85p9meE0MbIbX\n8lSWoEgH2hZZjgd2FlmuD/QEFoZP0rQC5pvZCGA08KG75wGZZvY1kAD0A0aY2Q+A2kADM3sDGAM0\nMrPq4aOKUx/rW+4+EZgIkJCQUGyYiIiUh2M5+fzpsw289tUWYmtV59lRvbg5IdgZXstTWc5RLAG6\nhq9GqgncAsw/eaO7Z7l7M3fv4O4dgEXACHdPBrYDQy0kFhgErHX3x9w9Ptz+FuAzd7/d3R34HLgx\nfPdjgXnnp6siIufG3fkwbReXv/gFf/1iM6P6x/HZg//GrYntKm1IQBmOKNw938zuBT4CYoAp7r7K\nzJ4Ckt19fgmbTwCmAmmEhpWmuvuKUh7yEWCmmf07sBR4rQz9EBEpV9v2H+N381exMDzD63+N7seA\n9pEzw2t5stCL+OiWkJDgycnJQZchIpVQdl4B//3FJv68cBM1qhkPXNmNsd9pT/UIm+H1bJhZirsn\nlNZO78wWETmNhesy+d38VWzbf5xr+7ThieEX0bJBZM7wWp4UFCIip9iVdYKn31/NByt306lZLG+M\nG8jgrs2CLiswCgoRkbC8gkKmfr2Flz/ZQEGh89CVFzD+0uiY4bU8KShERIDFWw7wxNyVrN9zlO9f\n2ILfj+hB2ybRM8NreVJQiEiVtu9oDs9+sJbZqaEZXif9KIErurcMuqyIoqAQkSqpoNCZsXg7//Hh\nWk7kFfCzIZ25d2gX6tbU0+Kp9BsRkSpnRfohnpibxor0LL7buSlPjexJlxb1gi4rYikoRKTKyDqe\nx3/8fS1vJm2nWb1avHJLX0b0aROVnxFRkRQUIlLpuTtzUkMzvB48XjlneC1PCgoRqdTW7T7Cb+al\nsXjLAfq1a8T0OyvnDK/lSUEhIpXSsZx8Xvl0A1P+sYV6tavz3Khe/LASz/BanhQUIlKphGZ43c1T\n769mV1Y2Nye05ZGrL6RJbM2gS4taCgoRqTS27gvN8PrF+r1c1LpBlZrhtTwpKEQk6mXnFfCXhZv4\nyxebqBlTjd9e050fVZIZXiOBgkJEoppmeC1/CgoRiUo7D4VmeF2QtptOzWN58ycDuaRL1Z3htTwp\nKEQkquQVFDLlH1t45dMNFLrzq6u68ZPvdazyM7yWJwWFiESNpM37+c28NNbvOcrlF7Xkd9d21wyv\nFUBBISIRb++RHJ5dsIY5qRma4TUACgoRiVgFhc6MpG08/9E6svMKuOeyztx7WVfq1NQwU0VSUIhI\nRFq+IzTD68qMLC7p0pQnR2iG16AoKEQkopw6w+ufbu3Htb1ba4bXACkoRCQiuDuzUzN4NjzD64+/\n24H7r9AMr5FAQSEigVu3+wi/mZvG4q0H6N+uEa+PS6RHG83wGikUFCISmJMzvL72jy3Ur12dP97Q\ni5sGaIbXSKOgEJEK5+4sSNvNU++tZvfhbG65uC0PD9MMr5FKQSEiFWpLeIbXL9fvpXvrBky4rT8D\n2jcOuiwpgYJCRCpEdl4Bf164if9euIla1avxu2u7M2aQZniNBgoKESl3n6/L5HfzVrH9wHFG9m3D\n4z+4iBaa4TVqKChEpNzsPHSCp95bzYerQjO8zvjJQL6rGV6jjoJCRM673PxCpny9hVc+2YATmuF1\n/Pc6UbO6hpmiUZn2mpkNM7N1ZrbRzB4tod2NZuZmlhBermFm081spZmtMbPHwutrm9liM1tuZqvM\n7Mki9zHNzLaY2bLwV99z7aSIVJxFm/cz/E9f8dyCtQzu2oyP7/837rmsi0IiipV6RGFmMcAE4Aog\nHVhiZvPdffUp7eoDvwCSiqy+Cajl7r3MrC6w2szeArYBQ939qJnVAP5hZgvcfVF4u1+5+6xz7p2I\nVJi9R3J49oM1zFmaQXzjOkz+UQKXa4bXSqEsQ0+JwEZ33wxgZjOBkcDqU9o9DTwPPFRknQOxZlYd\nqAPkAofd3YGj4TY1wl9+tp0QkeAUFDpvJm3jP8IzvN57WRfuuayLZnitRMpyLBgH7CiynB5e9y0z\n6we0dff3T9l2FnAM2AVsB15w9wPhbWLMbBmQCXzs7kWPRP5gZivM7CUzq1VcUWZ2l5klm1ny3r17\ny9ANETnflu04xMgJ/+C381bRO74hH953KQ9d1U0hUcmUJSiKey/9t6/+zawa8BLwYDHtEoECoA3Q\nEXjQzDoBuHuBu/cF4oFEM+sZ3uYx4ELgYqAJ8EhxRbn7RHdPcPeE5s2bl6EbInK+HDqey+PvruT6\nP39N5uEcXr21H2+MG0jn5poGvDIqy9BTOtC2yHI8sLPIcn2gJ7AwPA1wK2C+mY0ARgMfunsekGlm\nXwMJwOaTG7v7ITNbCAwD0tx9V/imHDObyj8PZYlIgAoLndmp6Ty7YC1ZJ/K447sduf+KrtTXDK+V\nWlmOKJYAXc2so5nVBG4B5p+80d2z3L2Zu3dw9w7AImCEuycTGm4aaiGxwCBgrZk1N7NGAGZWB7gc\nWBtebh3+bsB1QNp56quInIO1uw9z88T/x69mraBjs1jeu3cwv722u0KiCij1iMLd883sXuAjIAaY\n4u6rzOwpINnd55ew+QRgKqEnewOmuvsKM+sNTA9fUVUNeKfI+Y03zax5uP0y4P+cbedE5Ny5OxM+\n38hLn2ygQe3qPH9Db24cEK8ZXqsQC12AFN0SEhI8OTk56DJEKh1357kFa/nrl5sZ0acNT47oQWPN\n8FppmFmKuyeU1k7vzBaRYrk7T763mmnfbGXMoPY8OaKHjiKqKAWFiPyLwkLniXlpzEjazrjBHXli\n+EX6zOoqTEEhIv+koNB5ZPYKZqWk89MhnXn4qm4KiSpOQSEi38ovKOSBd5Yzf/lO7ru8K7/8fleF\nhCgoRCQkN7+QX85cyoK03Tw8rBs/G9Il6JIkQigoRISc/ALueTOVT9Zk8sTwi/jJ9zoFXZJEEAWF\nSBWXnVfAXX9L4cv1e3l6ZA/GfKdD0CVJhFFQiFRhx3PzGTctmUVb9vPHG3px88Xtgi5JIpCCQqSK\nOpKdx53TlpCy7SAv/rAP1/eLD7okiVAKCpEqKOtEHmOnLGZlRhZ/urUf1/RuE3RJEsEUFCJVzMFj\nuYyZksS63Uf48239uapHq6BLkginoBCpQvYdzeH2yUls3neMiWMSuOzCFkGXJFFAQSFSRWQezmb0\n5CTSDx5nytiLGdy1WdAlSZRQUIhUATsPnWD0pEVkHslh2h2JDOrUNOiSJIooKEQquR0HjnPrpEVk\nHc/jb+MSGdC+SdAlSZRRUIhUYlv2HeO2SYs4llvAm+MH0ju+UdAlSRRSUIhUUhszjzB6UhL5hc6M\n8QPp0aZh0CVJlFJQiFRCa3cf5vbJSYAx865BXNCyftAlSRSrFnQBInJ+pWVkccvERcRUM96+WyEh\n505HFCKVyNLtBxk7ZTH1a9dgxviBtG8aG3RJUgkoKEQqiSVbD3DH1CU0ia3JjPEDiW9cN+iSpJJQ\nUIhUAt9s2sdPpifTqkFtZowfRKuGtYMuSSoRnaMQiXJfrN/LHVOXENeoDjPvVkjI+acjCpEo9uma\nPfz0jVQ6t6jHG+MSaVqvVtAlSSWkoBCJUgtW7uLnby2le5sGvH5nIo3q1gy6JKmkFBQiUWjesgwe\neGc5feIbMu3ORBrUrhF0SVKJKShEosyslHQenrWchA5NmPLji6lXS//GUr70FyYSRWYkbefxuSu5\npHMzJv0ogTo1Y4IuSaoAXfUkEiWmfb2FX7+7kiEXNGfyWIWEVBwdUYhEgYlfbuKZD9ZyZfeWvDq6\nH7WqKySk4pTpiMLMhpnZOjPbaGaPltDuRjNzM0sIL9cws+lmttLM1pjZY+H1tc1ssZktN7NVZvZk\nkfvoaGZJZrbBzN42M13KIVXaf322gWc+WMvw3q2ZcFt/hYRUuFKDwsxigAnA1UB34FYz615Mu/rA\nL4CkIqtvAmq5ey9gAHC3mXUAcoCh7t4H6AsMM7NB4W3+CLzk7l2Bg8C4s+uaSHRzd/7z7+t44e/r\nub5fHK/c3JcaMRotlopXlr+6RGCju29291xgJjCymHZPA88D2UXWORBrZtWBOkAucNhDjobb1Ah/\nuZkZMBSYFb5tOnDdGfZJJOq5O88tWMurn23k5oS2vHBTH6orJCQgZfnLiwN2FFlOD6/7lpn1A9q6\n+/unbDsLOAbsArYDL7j7gfA2MWa2DMgEPnb3JKApcMjd80/3WCKVnbvz5Hur+euXmxkzqD3PjupF\nTDULuiypwsoSFMX9hfq3N5pVA14CHiymXSJQALQBOgIPmlknAHcvcPe+QDyQaGY9S3usfyrK7C4z\nSzaz5L1795ahGyKRr7DQeXxuGtO+2cq4wR15amQPqikkJGBlCYp0oG2R5XhgZ5Hl+kBPYKGZbQUG\nAfPDJ7RHAx+6e567ZwJfAwlF79zdDwELgWHAPqBReKiquMcqut1Ed09w94TmzZuXoRsika2g0Hl4\n9gpmJG3np0M688TwiwiNxooEqyxBsQToGr4aqSZwCzD/5I3unuXuzdy9g7t3ABYBI9w9mdBw01AL\niSUUImvNrLmZNQIwszrA5cBad3fgc+DG8N2PBeadl56KRLD8gkLuf3sZs1LSue/yrjx8VTeFhESM\nUoMifL7gXuAjYA3wjruvMrOnzGxEKZtPAOoBaYQCZ6q7rwBaA5+b2Yrw+o+LnN94BHjAzDYSOmfx\n2ln0SyRq5OYX8vO3ljJ/+U4eHtaN+y6/QCEhEcVCL+KjW0JCgicnJwddhsgZy8kv4J43U/lkTSa/\nuaY74wZ3DLokqULMLMXdE0prp3dmiwQkO6+Au/6Wwpfr9/L0dT0ZM6h90CWJFEtBIRKA47n5jJuW\nzKIt+/njDb24+eJ2QZckcloKCpEKdiQ7jzunLSFl20Fe/GEfru8XH3RJIiVSUIhUoKwTeYydspi0\njCxevbU/w3u3DrokkVIpKEQqyMFjuYyZksS63Uf48239ubJHq6BLEikTBYVIBdh3NIfbJyexed8x\nJo5J4LILWwRdkkiZKShEylnm4WxGT04i/eBxpoy9mMFdmwVdksgZUVCIlKOdh04wetIi9h7JYfod\niQzs1DTokkTOmIJCpJzsOHCcWyctIut4Hq+PG8iA9o2DLknkrCgoRMrBln3HuG3SIo7lFvDm+IH0\njm8UdEkiZ01BIXKebcw8wuhJSeQXOjPGD6RHm4ZBlyRyThQUIufR2t2HuX1yEmDMvGsQF7SsH3RJ\nIudMn60ocp6kZWRxy8RFxFQz3r5bISGVh44oRM6DpdsPMnbKYurXrsGM8QNp3zQ26JJEzhsFhcg5\nWrL1AHdMXUKT2JrMGD+Q+MZ1gy5J5LxSUIicg2827eMn05Np1aA2M8YPolXD2kGXJHLe6RyFyFn6\nYv1e7pi6hPjGdZh5t0JCKi8dUYichU/X7OGnb6TSuUU93hiXSNN6tYIuSaTcKChEztCClbv4+VtL\n6d6mAa/fmUijujWDLkmkXCkoRM7AvGUZPPDOcvrEN2TanYk0qF0j6JJEyp3OUYiU0ayUdO5/exkJ\n7Rvz+riBCgmpMnREIVIGM5K28/jclVzSuRmTfpRAnZoxQZckUmF0RCFSimlfb+HX765kyAXNmTxW\nISFVj44oREow8ctNPPPBWq7s3pJXR/ejVnWFhFQ9CgqR0/ivzzbwwt/XM7x3a16+uS81YnQALlWT\ngkLkFO7Oix+v59XPNjKqXxzP39ib6goJqcIUFCJFuDvPLVjLX7/czM0JbXlmVC9iqlnQZYkESkEh\nEubuPPneaqZ9s5Uxg9rz5IgeVFNIiCgoRAAKC50n5qUxI2k74wZ35InhF2GmkBABBYUIBYXOI7NX\nMCslnZ8N6cyvruqmkBApokxn6MxsmJmtM7ONZvZoCe1uNDM3s4Twcg0zm25mK81sjZk9Fl7f1sw+\nD69bZWa/LHIfvzezDDNbFv76wbl2UuR08gsKuf/tZcxKSee+y7sqJESKUeoRhZnFABOAK4B0YImZ\nzXf31ae0qw/8AkgqsvomoJa79zKzusBqM3sLyAEedPfU8HYpZvZxkft8yd1fOOfeiZQgN7+QX85c\nyoK03Tw8rBs/G9Il6JJEIlJZjigSgY3uvtndc4GZwMhi2j0NPA9kF1nnQKyZVQfqALnAYXff5e6p\nAO5+BFgDxJ19N0TOTE5+AT97M4UFabv5zTXdFRIiJShLUMQBO4osp3PKk7qZ9QPauvv7p2w7CzgG\n7AK2Ay+4+4FTtu0A9OOfj0TuNbMVZjbFzBqXoUaRMsvOK2D86yl8siaTp6/rybjBHYMuSSSilSUo\nihuw9W9vNKsGvAQ8WEy7RKAAaAN0BB40s05Ftq0HzAbuc/fD4dV/AToDfQkFzH8WW5TZXWaWbGbJ\ne/fuLUM3ROB4bj53TF3CVxv28vwNvRkzqH3QJYlEvLIERTrQtshyPLCzyHJ9oCew0My2AoOA+eET\n2qOBD909z90zga+Bb090EwqJN919zsk7c/c97l7g7oXAJEJh8y/cfaK7J7h7QvPmzcvWW6nSjmTn\nMXbKYpK27OfFH/bhhxe3LX0jESlTUCwBuppZRzOrCdwCzD95o7tnuXszd+/g7h2ARcAId08mNNw0\n1EJiCYXIWgtdVvIasMbdXyz6YGbWusji9UDaOfRPBICsE3mMeW0xS7cf4tVb+3N9v/igSxKJGqUG\nhbvnA/cCHxE66fyOu68ys6fMbEQpm08A6hF6sl8CTHX3FcAlwBhCIXLqZbDPhy+nXQFcBtx/Vj0T\nCTt4LJfbJi9i1c4s/nxbf4b3bl36RiLyLXP30ltFuISEBE9OTg66DIlA+47mcPvkJDbvO8Zfxwzg\nsm4tgi5JJGKYWYq7J5TWTu/Mlkor83A2oycnkX7wOFPGXszgrs2CLkkkKikopFLaeegEoyctYu+R\nHKbfkcjATk2DLkkkaikopNLZceA4t05aRNbxPF4fN5AB7fVWHJFzoaCQSmXLvmPcNmkRx3ILeHP8\nQHrHNwq6JJGop6CQSmNj5hFGT0oiv9CZMX4gPdo0DLokkUpBQSGVwtrdh7l9chJgzLxrEBe0rB90\nSSKVhj4IWKJeWkYWt0xcRPVq1XjnboWEyPmmIwqJaku3H2TslMXUr12DGeMH0r5pbNAliVQ6CgqJ\nWku2HuCOqUtoEluTGeMHEt+4btAliVRKCgqJSt9s2sdPpifTqkFtZowfRKuGtYMuSaTS0jkKiTpf\nrN/LHVOXEN+4DjPvVkiIlDcdUUhU+XTNHn76RiqdW9TjjXGJNK1XK+iSRCo9BYVEjQUrd/Hzt5bS\nvU0DXr8zkUZ1awZdkkiVoKCQqDBvWQYPvLOcPvENmXZnIg1q1wi6JJEqQ+coJOLNSknn/reXkdC+\nMa+PG6iQEKlgOqKQiDYjaTuPz13J4C7NmDgmgTo1Y4IuSaTK0RGFRKxpX2/h1++uZMgFzZn0I4WE\nSFB0RCERaeKXm3jmg7Vc2b0lr47uR63qCgmRoCgoJKLk5hfyl4WbeOmT9Qzv3ZqXb+5LjRgd+IoE\nSUEhgXN30jIOMzs1nfnLd3LgWC6j+sXx/I29qa6QEAmcgkICszsrm7nLMpidks6GzKPUrF6NK7q3\n5Ib+cQy5oAXVqlnQJYoICgqpYCdyC/ho1W5mp6bz9cZ9FDoMaN+YZ67vxfBerWlYV5e+ikQaBYWU\nu8JCZ/HWA8xJTeeDlbs5mpNPXKM63HtZF67vH0/HZpoaXCSSKSik3Gzdd4w5qenMWZpB+sETxNaM\n4Qe9WnPDgHgSOzTR0JJIlFBQyHmVdSKP/1mxi9mp6aRsO4gZDO7SjIeu7MZVPVrpvRAiUUhBIecs\nv6CQLzfsZXZqBh+v3kNufiFdW9Tj0asv5Lq+cZoGXCTKKSjkrK3eGbqkdd6ynew7mkPjujUYndiO\nUf3j6BXXEDMNLYlUBgoKOSOZR7KZv2wns1MzWLPrMDVijO9f2JJR/eMY0q0FNavrfQ8ilY2CQkqV\nnVfAJ2v2MDslnS837KOg0OnTthFPjezBtb3b0DhWnwshUpkpKKRY7k7KtoPMTs3g/RU7OZKdT+uG\ntbn70k6M6h9Hlxb1gy5RRCqIgkL+yY4Dx5mTmsGcpels23+cOjViuLpnK24YEM+gTk2J0SWtIlVO\nmYLCzIYBrwAxwGR3f+407W4E/i9wsbsnm1kNYDLQP/xYr7v7s2bWFngdaAUUAhPd/ZXwfTQB3gY6\nAFuBH7r7wbPuoZTqSHYeC1buZlZqOou3HMAMvtOpKT8f2pWre7YitpZeT4hUZaU+A5hZDDABuAJI\nB5aY2Xx3X31Ku/rAL4CkIqtvAmq5ey8zqwusNrO3gBzgQXdPDW+XYmYfh+/zUeBTd3/OzB4NLz9y\n7l2VogoKna837mN2ajofrdpNdl4hnZrF8qurunFdvzjiGtUJukQRiRBleamYCGx0980AZjYTGAms\nPqXd08DzwENF1jkQa2bVgTpALnDY3Q8AuwDc/YiZrQHiwvc5EhgS3n46sBAFxXmzfs8RZqemM3dp\nBnsO59CwTg1uHBDPqP7x9GvbSJe0isi/KEtQxAE7iiynAwOLNjCzfkBbd3/fzIoGxSxCT/y7gLrA\n/eGQKLptB6Af/3sk0tLdT4bILjNrUVxRZnYXcBdAu3btytCNqmv/0RzmL9/JnNQMVmZkUb2aMaRb\nc35/bTxDL2qhDwUSkRKVJSiKe4np395oVg14CfhxMe0SgQKgDdAY+MrMPilydFIPmA3c5+6Hz6Rw\nd58ITARISEjwUppXOTn5BXy+NpNZKRksXJdJfqHTo00DfntNd0b0bUOzerWCLlFEokRZgiIdaFtk\nOR7YWWS5PtATWBgetmgFzDezEcBo4EN3zwMyzexrIAHYHD7RPRt4093nFLm/PWbWOnw00RrIPMu+\nVTnuzrIdh5iTmsF7K3Zy6HgeLerXYtzgjlzfP44LWzUIukQRiUJlCYolQFcz6whkALcQCgAA3D0L\naHZy2cwWAg+Fr3r6PjDUzN4gNPQ0CHjZQonyGrDG3V885fHmA2OB58Lf551l36qMnYdO8O7SDGan\nprN57zFqVa/GVT1Cl7Re0rmpPiVORM5JqUHh7vlmdi/wEaHLY6e4+yozewpIdvf5JWw+AZgKpBEa\nwprq7ivMbDAwBlhpZsvCbX/t7h8QCoh3zGwcsJ3QlVNyimM5+XyYtps5S9P5ZtN+3CGxQxPuvrQT\nV/dqTYPa+gAgETk/zD36h/cTEhI8OTk56DLKXWGhs2jzfmalpvNh2m6O5xbQrkldRvWPY1S/eNo1\nrRt0iSISRcwsxd0TSmund1JFgU17jzInNZ13UzPYmZVN/VrVGdm3DaP6x5PQvrEuaRWRcqWgiFCH\njufy3vLQLK3LdhyimsGlFzTnsR9cxBXdW1K7hi5pFZGKoaCIIHkFhSxct5fZKel8tjaT3IJCLmxV\nn8d/cBEj+7ahRQN9AJCIVDz4i7OkAAAGS0lEQVQFRcDcnVU7DzMrJZ35y3dy4FguTWNrcvug9tww\nII7urRtoaElEAqWgCMiew9nMDV/Sun7PUWrGVOOK7qEPALr0gubU0CWtIhIhFBQV6ERuAX9fvZvZ\nqRn8Y8NeCh36t2vEH67vyTW92tCwri5pFZHIo6AoZ4WFzpKtB5idms4HK3dzNCefuEZ1uOeyLozq\nH0/HZrFBlygiUiIFRTnZtv8Ys1MzeHdpOjsOnCC2ZgxX92rNDf3jGdixCdX0AUAiEiUUFOdR1ok8\nPli5i9kp6SRvO4gZDO7SjAeuuICrerSibk39ukUk+uiZ6xzlFxTy1YbQBwD9ffUecvML6dKiHo8M\nu5Dr+rWhdUN9AJCIRDcFxVlas+sws1PSmbtsJ/uO5tC4bg1uvbgtNwyIp1dcQ13SKiKVhoLiDOw9\nksO8ZRnMTs1gza7D1IgxLuvWghsGxHNZtxbUrK5LWkWk8lFQlCI7r4BP12QyOzWdL9bvpaDQ6RPf\nkCdH9ODaPm1oElsz6BJFRMqVgqIY7k7q9oPMTs3g/eU7OZydT6sGtbnr0k6M6hdH15b1gy5RRKTC\nKCiK2HHgOO8uzWBOajpb9x+nTo0YhvVsxQ394/lO56bE6JJWEamCqnxQHM3J//aS1qQtBwD4Tqem\n3HNZF67u1Zp6tar8r0hEqrgq/Sz4p0838OeFG8nOK6Rjs1gevOICru8fR3xjfQCQiMhJVTooWjes\nzQ394xnVP57+7RrpklYRkWJU6aC4KaEtNyW0DboMEZGIpgv/RUSkRAoKEREpkYJCRERKpKAQEZES\nKShERKRECgoRESmRgkJEREqkoBARkRKZuwddwzkzs73AtrPcvBmw7zyWEyT1JfJUln6A+hKpzqUv\n7d29eWmNKkVQnAszS3b3hKDrOB/Ul8hTWfoB6kukqoi+aOhJRERKpKAQEZESKShgYtAFnEfqS+Sp\nLP0A9SVSlXtfqvw5ChERKZmOKEREpERVKijMbIqZZZpZWpF1TczsYzPbEP7eOMgay+o0ffm9mWWY\n2bLw1w+CrLEszKytmX1uZmvMbJWZ/TK8Pur2Swl9icb9UtvMFpvZ8nBfngyv72hmSeH98raZ1Qy6\n1pKU0I9pZralyD7pG3StZWVmMWa21MzeDy+X+z6pUkEBTAOGnbLuUeBTd+8KfBpejgbT+Ne+ALzk\n7n3DXx9UcE1nIx940N0vAgYB95hZd6Jzv5yuLxB9+yUHGOrufYC+wDAzGwT8kVBfugIHgXEB1lgW\np+sHwK+K7JNlwZV4xn4JrCmyXO77pEoFhbt/CRw4ZfVIYHr45+nAdRVa1Fk6TV+ijrvvcvfU8M9H\nCP0DxBGF+6WEvkQdDzkaXqwR/nJgKDArvD7i90sJ/YhKZhYPDAcmh5eNCtgnVSooTqOlu++C0D86\n0CLges7VvWa2Ijw0FfHDNUWZWQegH5BElO+XU/oCUbhfwkMcy4BM4GNgE3DI3fPDTdKJgiA8tR/u\nfnKf/CG8T14ys1oBlngmXgYeBgrDy02pgH2ioKhc/gJ0JnSIvQv4z2DLKTszqwfMBu5z98NB13Mu\niulLVO4Xdy9w975APJAIXFRcs4qt6syd2g8z6wk8BlwIXAw0AR4JsMQyMbNrgEx3Tym6upim532f\nKChgj5m1Bgh/zwy4nrPm7nvC/xSFwCRC/9wRz8xqEHpifdPd54RXR+V+Ka4v0bpfTnL3Q8BCQudd\nGplZ9fBN8cDOoOo6U0X6MSw8TOjungNMJTr2ySXACDPbCswkNOT0MhWwTxQUMB8YG/55LDAvwFrO\nyckn1rDrgbTTtY0U4THW14A17v5ikZuibr+cri9Rul+am1mj8M91gMsJnXP5HLgx3Czi98tp+rG2\nyIsQIzSmH/H7xN0fc/d4d+8A3AJ85u63UQH7pEq94c7M3gKGEJptcQ/wO2Au8A7QDtgO3OTuEX+S\n+DR9GUJoeMOBrcDdJ8f5I5WZDQa+Albyv+OuvyY0th9V+6WEvtxK9O2X3oROjMYQekH5jrs/ZWad\nCL2abQIsBW4PvyqPSCX04zOgOaGhm2XA/yly0jvimdkQ4CF3v6Yi9kmVCgoRETlzGnoSEZESKShE\nRKRECgoRESmRgkJEREqkoBARkRIpKEREpEQKChERKZGCQkRESvT/AeJBXRppDxvrAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ad70c10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.49137832, -0.48948976, -0.4904535 , -0.49047327])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 再往下细调"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n",
      "[CV] min_child_samples=1 .............................................\n",
      "[CV] min_child_samples=1 .............................................\n",
      "[CV] min_child_samples=1 .............................................\n",
      "[CV] min_child_samples=3 .............................................\n",
      "[CV] ....... min_child_samples=3, score=-0.474089089112, total= 1.6min\n",
      "[CV] min_child_samples=3 .............................................\n",
      "[CV] ....... min_child_samples=1, score=-0.485457980263, total= 1.6min\n",
      "[CV] min_child_samples=3 .............................................\n",
      "[CV] ....... min_child_samples=1, score=-0.470505084313, total= 1.6min\n",
      "[CV] min_child_samples=5 .............................................\n",
      "[CV] ....... min_child_samples=1, score=-0.489063071721, total= 1.6min\n",
      "[CV] min_child_samples=5 .............................................\n",
      "[CV] ....... min_child_samples=3, score=-0.488772781703, total= 1.6min\n",
      "[CV] min_child_samples=5 .............................................\n",
      "[CV] ....... min_child_samples=3, score=-0.485345543887, total= 1.7min\n",
      "[CV] ....... min_child_samples=5, score=-0.472974524086, total= 1.7min\n",
      "[CV] min_child_samples=7 .............................................\n",
      "[CV] min_child_samples=7 .............................................\n",
      "[CV] ....... min_child_samples=5, score=-0.487588163019, total= 1.7min\n",
      "[CV] min_child_samples=7 .............................................\n",
      "[CV] ....... min_child_samples=7, score=-0.474777193148, total= 1.6min\n",
      "[CV] min_child_samples=9 .............................................\n",
      "[CV] ....... min_child_samples=7, score=-0.491373272154, total= 1.6min\n",
      "[CV] ....... min_child_samples=7, score=-0.485861153925, total= 1.6min\n",
      "[CV] min_child_samples=9 .............................................\n",
      "[CV] min_child_samples=9 .............................................\n",
      "[CV] ....... min_child_samples=5, score=-0.487530757419, total= 1.6min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  6.8min remaining:  1.7min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ....... min_child_samples=9, score=-0.474541358025, total= 1.1min\n",
      "[CV] ....... min_child_samples=9, score=-0.484814648141, total= 1.2min\n",
      "[CV] ....... min_child_samples=9, score=-0.491693844401, total= 1.2min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:  8.4min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', colsample_bytree=0.7, learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=10, min_child_weight=5,\n",
       "        min_split_gain=0.0, n_estimators=374, n_jobs=4, num_class=9,\n",
       "        num_leaves=70, objective='multiclass', random_state=0,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=50000, subsample_freq=1),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'min_child_samples': [1, 3, 5, 7, 9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(1,10,2) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.481674778941\n",
      "{'min_child_samples': 1}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VdW5//HPkwkS5iEMCUMQQUSm\nSMSp4kjFCbAOP7V1qFZKb60T1qHt9VbtpHW6tlbrFS3OIipQarFqsagtSIAkzBARIQkQkHnI/Pz+\nOAc8YsjAkH1O8n2/XnmRvc/a5zwbwvlmrbXP2ubuiIiIHEhc0AWIiEh0U1CIiEiNFBQiIlIjBYWI\niNRIQSEiIjVSUIiISI0UFCIiUiMFhYiI1EhBISIiNUoIuoDDoWPHjp6RkRF0GSIiMWXevHmb3D21\ntnaNIigyMjLIzs4OugwRkZhiZl/UpZ2GnkREpEYKChERqZGCQkREaqSgEBGRGikoRESkRnUKCjMb\naWbLzSzfzO6uod2lZuZmlhXeTjSziWa20MyWmtk9+7WPN7MFZjY9Yl8vM5tjZivN7HUzSzrYkxMR\nkUNXa1CYWTzwJHAe0B+40sz6V9OuFXAzMCdi92VAM3cfCAwFfmhmGRGP3wIs3e+pHgQec/c+wBbg\nhrqejIiIHH516VEMA/LdfZW7lwGvAaOrafcA8BBQErHPgRZmlgAkA2XAdgAz6wZcADy7t7GZGXAW\nMDm8ayIwpj4nJCKxaUdJOc99/DnL1m8PuhTZT10+cJcOrI3YLgBOjGxgZplAd3efbmZ3RDw0mVCo\nrANSgNvcfXP4sceBO4FWEe07AFvdvSLitdLreC4iEsN+MWURU3OKAOjXpRVjMtMZPSSNrm2SA65M\n6hIUVs0+3/egWRzwGHBdNe2GAZVAGtAO+MjM3ic0hFXs7vPM7Iy6vtbXijIbC4wF6NGjR60nISLR\n629565iaU8S403uT1rY5by8o5Hd/X8aDM5ZxYq/2XJyZzsgBXWmTnBh0qU2SuVf7PvxVA7OTgV+6\n+7nh7XsA3P234e02wGfAzvAhXYDNwCjg+8Bsd38x3PY5YAaQCVwNVADNgdbAW+F9G4Eu7l6x/2sf\nSFZWlmsJD5HYVLy9hHMfn0WPDi14c9zJJMSHRsRXb9rF1JwipuYUsmrTLpIS4ji7XyfGZKZzxjGp\nNEuID7jy2Gdm89w9q9Z2dQiKBGAFcDZQCMwFrnL3xQdo/yFwh7tnm9ldQD/gekJDT3OBK9w9L6L9\nGeH2F4a33wDedPfXzOxpIM/d/1RTjQoKkdjk7twwMZtP8jfxzi2n0Tu1ZbVt8gq28faCQqbnFbFp\nZxmtmydwwaA0xgxJ44SM9sTFVTcYIbWpa1DUOvQU/s3+JuBdIB54zt0Xm9n9QLa7T6vh8CeB54FF\nhIaVno8MiQO4C3jNzH4FLAAm1FajiMSm1+eu5Z/LivnlRf2rDQkAM2Nw97YM7t6WX1xwLB/nb2Jq\nThFTFhTy6qdrSG+bzOghaYzJTKdv51bVPoccmlp7FLFAPQqR2LN2825GPj6LIT3a8uL1J9a7V7Cr\ntIL3lmxgSk4hH63cRGWV079ra8ZkpjFqcDpd2jQ/QpU3Hodt6CkWKChEYktllXPlM7NZum477942\nnLS2h3Zl06adpUzPLeLtnCJy127FDE4+qgNjMtMZOaALrZtrErw6CgoRiVr/N2sVv35nKY9cNphL\nhnY7rM/9+aZdTFlQyJScQr74cjfNEuI459jOjMlM5/S+qSQlaOWivRQUIhKVVmzYwYVPfMyZ/VJ5\n+ntDCX3O9vBzd3LWbmXKgkKm563jy11ltE1J5IKBXRmTmc7QHu2a/CS4gkJEok5ZRRUX/+kT1m8r\n4R+3DadDy2YN8rrllVV8vHITU3IKeXfxekrKq+jWLjQJfnFmOkd3apqT4IftqicRkcPlj/9cyeKi\n7Txz9dAGCwmAxPg4zuzXiTP7dWJXaQX/WLKetxcU8dSHn/HkzM8YkN6aMUPSGTU4jU6tNQm+P/Uo\nRKRBLFizhUuf/g8XZ6bz8GWDgy4HgOIdJUzPXceUnELyCrYRZ3BK746MyUzn3OM606qRT4Jr6ElE\nosaeskoueOIjSiuq+Putp0XlVUifbdzJ1AWFTMkpYs3m0CT4iP6dGTMkneGNdBJcQ08iEjUenLGM\nVZt28coPTozKkADondqS2799DLeN6Mv8NVuZmlPIX3OLmJ63jnYpiVw4KI0xmWkc36PdEZuAj1bq\nUYjIEfVJ/ia+++wcvn9qBv9z0XFBl1Mv5ZVVzFqxkSk5Rby3JDQJ3qN9CqOHpDF6SDpHd6r+0+Sx\nQkNPIhK47SXljHxsFslJ8fzt5tNonhi7C/ntLK3g3UXrmZJTyCf5m6hyGNStDaOHpHPR4K50ahV7\nk+AKChEJ3PhJuUzJKeStH53C4O5tgy7nsCneXsK03CKm5BSyqHA7cQanHt2RizPT+fZxXWjZLDZG\n9RUUIhKoGYvWM+6ledx8dh9uH9E36HKOmPziHUxZEAqNgi17aJ4Yx7f7d+HizHS+1acjifHROwmu\noBCRwGzaWcq5j82ia9vmvP1fp0b1m+Xh4u7M+2ILU3JCnwTfuruc9i2SuGhQV0ZnppPZvW3UTYIr\nKEQkEO7O2Bfn8a8VG/nbT75Fnya49HdZRWgS/O2cQt5fsoHSiip6dkhh9JB0xgxJ46gDLKne0HR5\nrIgE4s35hby3ZAO/uODYJhkSAEkJcZzTvzPn9O/MjpJyZoQnwf/wz5U88cFKBndrw5jMdC4clEZq\nq4b7hPrBUo9CRA6bgi27Oe/xj+if1ppXbzypyS+6t7/120r4a3gSfHHRduLjjG/tmwTvTEpSw/7u\nrqEnEWlQVVXOd5+dQ17BVmbcOpzu7VOCLimqrdywgyk5hUxZUETh1j0kJ8Zz7nGdGZ2ZzmlHd9x3\n7/AjSUNPItKgJv5nNf9Z9SUPXjJQIVEHfTq34qfn9mP8iGPIDk+C/y1vHVNyiujYMin8SfB0Bndr\nE/gkuHoUInLI8ot3csETH/Gtozvy7LVZgb+xxarSikr+tXwjU3IKeX9pMWUVVfTq2CJ0T/Ah6WR0\nbHFYX09DTyLSIMorq7jkqX+zdvNu3r1teEx+QjkabS8pZ8bC9by9oJDZn3+JOwzp3paLM9O5cFDX\nw7JMu4JCRBrE/76/ksfeX8Gfvns85w/sGnQ5jdK6bXuYllPElJwilq4LTYIP7xNaDn1E/4OfBFdQ\niMgRt7BgGxf/6RMuHNSVx6/IDLqcJmH5+tAk+NQFhRRtK+Hp7w1l5IAuB/VcCgoROaJKyiu58A8f\ns7OkgndvHU6blOhcPryxqqpyPl29mcwebWmWcHCLLeqqJxE5oh5+dzn5xTt54fphCokAxMUZJx3V\noWFeq0FeRUQaldmrvmTCJ59z9Uk9Gd43Nehy5AhTUIhIvewoKeeON3Lp2T6Fe87vF3Q50gA09CQi\n9fKr6Usp2rqHN8ad0uBLTkgw1KMQkTr7YOkGXs9ey7jTezO0Z7ugy5EGoqAQkTrZvKuMu95cyLFd\nW3PrOY33RkTyTeo3ikit3J2fv72QbXvKePGGYSQl6HfMpkT/2iJSq6k5Rfx90XpuH3EMx3ZtHXQ5\n0sAUFCJSo3Xb9nDv1EUM7dmOscOPCrocCYCCQkQOyN25c3Ie5ZXOI5cNJl43ImqS6hQUZjbSzJab\nWb6Z3V1Du0vNzM0sK7ydaGYTzWyhmS01s3vC+5ub2admlmtmi83svojn+IuZfW5mOeGvIYd6kiJy\ncF6a/QUfrdzEzy849rAvcS2xo9bJbDOLB54ERgAFwFwzm+buS/Zr1wq4GZgTsfsyoJm7DzSzFGCJ\nmb0KfAGc5e47zSwR+NjM/u7us8PH/dTdJx/y2YnIQft80y5+884yhvdN5bsn9gi6HAlQXXoUw4B8\nd1/l7mXAa8Doato9ADwElETsc6CFmSUAyUAZsN1DdobbJIa/Yn91QpFGoqKyivGTckhKiOOhSwbp\nRkRNXF2CIh1YG7FdEN63j5llAt3dffp+x04GdgHrgDXAw+6+OXxMvJnlAMXAe+4e2RP5tZnlmdlj\nZnbod+cQkXr586xVzF+zlftHH0eXNroRUVNXl6Co7leJfb/9m1kc8Bgwvpp2w4BKIA3oBYw3s6MA\n3L3S3YcA3YBhZjYgfMw9QD/gBKA9cFe1RZmNNbNsM8veuHFjHU5DROpiSdF2Hn9/BRcM6sqowWlB\nlyNRoC5BUQB0j9juBhRFbLcCBgAfmtlq4CRgWnhC+ypghruXu3sx8AnwtbXP3X0r8CEwMry9Ljw0\nVQo8TyhsvsHdn3H3LHfPSk3V6pUih0NpRSW3T8qhbUoSvxo9QENOAtQtKOYCfcysl5klAVcA0/Y+\n6O7b3L2ju2e4ewYwGxjl7tmEhpvOspAWhEJkmZmlmllbADNLBs4BloW3u4b/NGAMsOgwnauI1OKx\n91aybP0OHrpkEO1aJAVdjkSJWq96cvcKM7sJeBeIB55z98Vmdj+Q7e7Tajj8SUK9gkWEhrCed/c8\nMxsETAxfURUHTIqY33jZzFLD7XOAcQd7ciJSd9mrN/PnWZ9x5bDunNmvU9DlSBTRrVBFhF2lFZz/\nxEdUufP3W4bTspmWgWsKdCtUEamz37yzlDWbd/P62JMVEvINWsJDpImbubyYl+es4cbTjmJYr/ZB\nlyNRSEEh0oRt3V3GXZPz6Nu5JbeP0D0mpHrqY4o0YfdOXczmXWU8d90JNE+MD7ociVLqUYg0UX/N\nLWJabhG3nN2HAeltgi5HopiCQqQJKt5ewn9PXcTg7m350Rm9gy5HopyCQqSJcXfuejOPkvJKHr18\nMAnxehuQmuknRKSJeW3uWmYu38jdI/vRO7Vl0OVIDFBQiDQha77czQPTl3Dq0R245uSMoMuRGKGg\nEGkiKquc8W/kEB9n/P7SwcTptqZSR7o8VqSJmPDxKuau3sIjlw0mrW1y0OVIDFGPQqQJWL5+Bw+/\nu4Jzj+vMd45Pr/0AkQgKCpFGrqyiittez6F1cgK/uXig7jEh9aahJ5FG7g//XMmSddt55uqhdGip\nOwtL/alHIdKIzV+zhSdn5nPp0G58+7guQZcjMUpBIdJI7Smr5I5JuXRtk8y9F/UPuhyJYRp6Emmk\nHpyxjFWbdvHKjSfSunli0OVIDFOPQqQR+njlJv7y79V8/9QMTundMehyJMYpKEQamW17yvnp5Fx6\np7bgrpH9gi5HGgENPYk0MvdNW0zxjlLe+tEpuseEHBbqUYg0IjMWreOtBYX8+MyjGdy9bdDlSCOh\noBBpJDbuKOVnby9iQHprfnLW0UGXI42IgkKkEXB37nlrITtLK3js8iEk6h4Tchjpp0mkEZg8r4D3\nl27gznOPoU/nVkGXI42MgkIkxhVs2c19f13Cib3ac/2pvYIuRxohBYVIDKuqcu54Ixd35+HLdI8J\nOTIUFCIx7Pl/r2b2qs3ce1F/urdPCbocaaQUFCIxKr94Bw/NWMbZ/TpxeVb3oMuRRkxBIRKDyiur\nuH1SLilJ8fz2Et1jQo4sfTJbJAY9OTOfvIJtPPXd4+nUqnnQ5Ugjpx6FSIzJK9jKH/+Zz5ghaZw3\nsGvQ5UgToKAQiSEl5ZXcPimXji2bcd+oAUGXI02Ehp5EYsjv311OfvFOXrxhGG1SdI8JaRjqUYjE\niP989iUTPv6cq0/qyWl9UoMuR5qQOgWFmY00s+Vmlm9md9fQ7lIzczPLCm8nmtlEM1toZkvN7J7w\n/uZm9qmZ5ZrZYjO7L+I5epnZHDNbaWavm1nSoZ6kSKzbUVLOHW/kktEhhXvO1z0mpGHVGhRmFg88\nCZwH9AeuNLNv3IDXzFoBNwNzInZfBjRz94HAUOCHZpYBlAJnuftgYAgw0sxOCh/zIPCYu/cBtgA3\nHNypiTQeD0xfwrpte3jk8iGkJGnEWBpWXXoUw4B8d1/l7mXAa8Doato9ADwElETsc6CFmSUAyUAZ\nsN1DdobbJIa/3EIXg58FTA4/NhEYU89zEmlU3l+ygUnZBYw7vTdDe7YLuhxpguoSFOnA2ojtgvC+\nfcwsE+ju7tP3O3YysAtYB6wBHnb3zeFj4s0sBygG3nP3OUAHYKu7VxzotSJec6yZZZtZ9saNG+tw\nGiKx58udpdz9Vh7Hdm3Nref0DbocaaLqEhTVfeTT9z1oFgc8Boyvpt0woBJIA3oB483sKAB3r3T3\nIUA3YJiZDajttb620/0Zd89y96zUVE3sSePj7vz87UVs31PBo5cPJilB155IMOryk1cARC4k0w0o\nithuBQwAPjSz1cBJwLTwhPZVwAx3L3f3YuATICvyyd19K/AhMBLYBLQND1VV91oiTcaUnEJmLF7P\n7d/uy7FdWwddjjRhdQmKuUCf8NVIScAVwLS9D7r7Nnfv6O4Z7p4BzAZGuXs2oeGmsyykBaEQWWZm\nqWbWFsDMkoFzgGXu7sBM4NLw018LTD0sZyoSQ9Zt28O9UxeT1bMdN552VNDlSBNXa1CE5wtuAt4F\nlgKT3H2xmd1vZqNqOfxJoCWwiFDgPO/ueUBXYKaZ5YX3vxcxv3EXcLuZ5ROas5hwEOclErOqqpyf\nvpFHZZXzyOWDidc9JiRgdbrOzt3fAd7Zb9+9B2h7RsT3OwldIrt/mzwg8wDHryI0tyHSJL005ws+\nzt/Ery8eQM8OLYIuR0SfzBaJJqs27uQ37yzl9L6pXDWsR9DliAAKCpGoUVFZxfg3cmmWEM+DlwzS\nPSYkaugjniJR4s+zVrFgzVaeuDKTLm10jwmJHupRiESBxUXbePz9FVwwqCujBqcFXY7I1ygoRAJW\nWlHJ7a/n0jYliV+N1j0mJPpo6EkkYI++t4LlG3bw/HUn0K6FFkuW6KMehUiA5q7ezDOzVnHlsO6c\n2a9T0OWIVEtBIRKQXaUVjJ+US7d2yfz8gm+s3C8SNTT0JBKQX7+zlLVbdvP62JNp2Uz/FSV6qUch\nEoCZy4t5Zc4axp52FMN6tQ+6HJEaKShEGtjW3WXcNTmPvp1bctsI3WNCop/6uyIN7L+nLmbzrjKe\nu+4EmifGB12OSK3UoxBpQNNyi/hrbhG3ntOHAeltgi5HpE4UFCINZMP2Ev57yiKGdG/LuNN7B12O\nSJ0pKEQagLtz5+Q8SisqeeTywSTE67+exA79tIo0gFc/Xcu/VmzknvOOpXdqy6DLEakXTWZLnb01\nv4A/ffgZCXFGSlI8KUkJJCfFk5IUT3Ji/L7vU5ISSE4M7w8/Ftl27/697Rr7Hdy++HIXv/rbEk49\nugNXn9Qz6HJE6k1BIXXy8cpN/HRyHsd0bkV6u2T2lFWyp7ySL3eVsaesgt1llewpq2R3eSWVVV6v\n505KiAsFSOLXAyT5a6FSzf5wAEW2SY54jpSkeJolxAV6X4fKKmf8pFzi44zfXzqYuEYeitI4KSik\nVvnFO/nRy/Po06klk8bV/Clid6essoqSsip2l0cESFklu8sq9gXM1/aXV+z7fk/53v0VbN1dRtHW\nr+/fU15Zr9rN2BdAyUnxpCTu37NJIDkx7qseT2RYJcWRnJjwjV5QSlI8zcM9psRa5hqe/WgV2V9s\n4dHLB5PWNrletYtECwWF1GjzrjJumDiXZglxPHttVq1LTZgZzRLiaZYQTxsSD3s9VVVOSUXlNwOo\n/KvtvUGze2+4hHs6+/aH923dXU5J+defo7yyfr2hxHj7Ws+meeLXe0Ezl23k3OM6c3Fm+mH/uxBp\nKAoKOaDSikrGvTiPddtKeG3sSXRrlxJ0ScTFWfhN+cj86JZXVu0LklDPpyIikCrZE9FLOlAA7Smv\nZEdJBcXbSxnasx2/uXigbmsqMU1BIdVyd3721iI+Xb2ZP1yZyfE92gVdUoNIjI+jTXIcbZIPf29I\nJFbp8lip1lP/+ow35xdw2zl9uUi35hRp0hQU8g0zFq3joRnLGT0kjZvPPjrockQkYAoK+Zq8gq3c\n+noOx/doy4OXDNLYuogoKOQr67bt4QcTs+nYshnPXJOllU1FBFBQSNiu0gpu+Es2u8sqmXDtCXRs\n2SzokkQkSigohMoq55bXcli2fjt/vCqTY7q0CrokEYkiCgrhoRnLeH/pBv7nouM445hOQZcjIlFG\nQdHEvfbpGv48axXXnNyTa0/JCLocEYlCCoom7N+fbeIXUxYxvG8q917YP+hyRCRKKSiaqFUbd/Kj\nl+bTq2ML/nhVpm6kIyIHpHeHJmjr7jJumJhNQpzx3HUn0Lq5lqsQkQOrU1CY2UgzW25m+WZ2dw3t\nLjUzN7Os8HaimU00s4VmttTM7gnv725mM8P7FpvZLRHP8UszKzSznPDX+Yd6kvKVsooqxr00j8It\ne3jmmqF0bx/8Qn8iEt1qXRTQzOKBJ4ERQAEw18ymufuS/dq1Am4G5kTsvgxo5u4DzSwFWGJmrwKl\nwHh3nx8+bp6ZvRfxnI+5+8OHfHbyNe7OL6YsZPaqzTz+/4YwtGf7oEsSkRhQlx7FMCDf3Ve5exnw\nGjC6mnYPAA8BJRH7HGhhZglAMlAGbHf3de4+H8DddwBLAS3Yf4Q9M2sVk7ILuPnsPozR/RFEpI7q\nEhTpwNqI7QL2e1M3s0ygu7tP3+/YycAuYB2wBnjY3Tfvd2wGkMnXeyI3mVmemT1nZk1jfesj7N3F\n6/ndjGVcOKgrt53TJ+hyRCSG1CUoqlsVbt9twMwsDngMGF9Nu2FAJZAG9ALGm9lREce2BN4EbnX3\n7eHdTwG9gSGEAuaRaosyG2tm2WaWvXHjxjqcRtO1qHAbt76Ww+BubXn4ssFa6E9E6qUuQVEAdI/Y\n7gYURWy3AgYAH5rZauAkYFp4QvsqYIa7l7t7MfAJsG+im1BIvOzub+19Mnff4O6V7l4F/B+hsPkG\nd3/G3bPcPSs1NbVuZ9sErd9Wwg0T59K+RRLPXDNUC/2JSL3VJSjmAn3MrJeZJQFXANP2Puju29y9\no7tnuHsGMBsY5e7ZhIabzrKQFoRCZJmFfqWdACx190cjX8zMukZsXgwsOoTza9J2l1XwgxfmsrOk\ngmevzaJTq+ZBlyQiMajWoHD3CuAm4F1Ck86T3H2xmd1vZqNqOfxJoCWhN/u5wPPungecClxNKET2\nvwz2ofDltHnAmcBtB3VmTVxVlXP767ksKdrOH67K5NiurYMuSURilLl77a2iXFZWlmdnZwddRlR5\ncMYynvrwM+69sD/Xf6tX0OWISBQys3nunlVbO30yuxGalL2Wpz78jO+e2IPvn5oRdDkiEuMUFI3M\n7FVf8vO3F3Jan478ctRxusJJRA6ZgqIRWb1pF+NemkeP9in88arjSdRCfyJyGOidpJHYtruc6/8y\nFwOeu+4E2iRroT8ROTxqXetJol95ZRU/enkea7fs5uUfnETPDi2CLklEGhEFRYxzd+6duph/f/Yl\nj1w2mGG9tNCfiBxeGnqKcRM+/pxXP13Dj8/szSVDuwVdjog0QgqKGPb+kg38+p2lnD+wC+NHHBN0\nOSLSSCkoYtTiom3c/NoCBqa34ZHLhhAXp8tgReTIUFDEoOLtJfxgYjZtkhN59poskpO00J+IHDma\nzI4xe8oqufGFbLbtKeeNcSfTqbUW+hORI0tBEUOqqpzxb+SQV7iNZ67O4ri0NkGXJCJNgIaeYsij\n763gnYXr+dl5xzKif+egyxGRJkJBESPeml/AH2fmc+Ww7vzgNK0GKyINR0ERA+au3szdby7klN4d\nuH/0AC30JyINSkER5b74chdjX8imW7tknvruUC30JyINTu86UWzbntBCfw5MuO4E2qRooT8RaXgK\niihVXlnFTa/MZ83m3Tz9vaH06qiF/kQkGLo8Ngq5O7+ctpiPVm7ioUsHcdJRHYIuSUSaMPUootDz\nn6zm5TlrGHd6by7P6h50OSLSxCkoosw/l23gV39bwrf7d+bOc7XQn4gET0ERRZat385PXllA/7TW\nPH6FFvoTkeigoIgSxTtKuOEv2bRsnsCz15xASpKmj0QkOujdKAqUlFcy9oV5bN5VxhvjTqZLGy30\nJyLRQ0ERsKoq5443cskt2MrT3xvKgHQt9Cci0UVDTwF7/IOVTM9bx10j+3HucV2CLkdE5BsUFAGa\nsqCQJz5YyeVZ3fjh8KOCLkdEpFoKioDM+2Izd07O48Re7fnVmIFa6E9EopaCIgBrN+9m7AvzSGvb\nnKe/N5SkBP0ziEj00jtUA9teElror7yyignXnUC7FklBlyQiUiNd9dSAKiqruOmVBXy+aRcvXD+M\n3qktgy5JRKRWCooG9MD0JcxasZHffWcgpxzdMehyRETqRENPDWTiv1cz8T9fcONpvbhiWI+gyxER\nqTMFRQP4cHkx9/11Mecc25m7zzs26HJEROqlTkFhZiPNbLmZ5ZvZ3TW0u9TM3MyywtuJZjbRzBaa\n2VIzuye8v7uZzQzvW2xmt0Q8R3sze8/MVob/bHeoJxmk5et3cNMrCzimS2v+94ohxGuhPxGJMbUG\nhZnFA08C5wH9gSvNrH817VoBNwNzInZfBjRz94HAUOCHZpYBVADj3f1Y4CTgxxHPeTfwgbv3AT4I\nb8ekTTtLuWHiXFKS4plwbRYtmmlKSERiT116FMOAfHdf5e5lwGvA6GraPQA8BJRE7HOghZklAMlA\nGbDd3de5+3wAd98BLAXSw8eMBiaGv58IjKnfKUWH0EJ/2WzaWcqz12aR1jY56JJERA5KXYIiHVgb\nsV3AV2/qAJhZJtDd3afvd+xkYBewDlgDPOzum/c7NgPI5KueSGd3XwcQ/rNTdUWZ2Vgzyzaz7I0b\nN9bhNBqOu3Pn5Dzmr9nKo5cPYVC3tkGXJCJy0OoSFNUNqvu+B83igMeA8dW0GwZUAmlAL2C8mR0V\ncWxL4E3gVnffXo+6cfdn3D3L3bNSU1Prc+gR98QH+UzLLeKn5x7D+QO7Bl2OiMghqUtQFACRN27u\nBhRFbLcCBgAfmtlqQnMO08IT2lcBM9y93N2LgU+AfRPdhELiZXd/K+L5NphZ13CbrkDxwZxYUKbl\nFvHY+yv4zvHp/NcZvYMuR0TkkNUlKOYCfcysl5klAVcA0/Y+6O7b3L2ju2e4ewYwGxjl7tmEhpvO\nspAWhEJkmYVWwJsALHX3R/dTbVGvAAAHZ0lEQVR7vWnAteHvrwWmHsL5Naj5a7Zwxxu5DMtoz2+/\no4X+RKRxqDUo3L0CuAl4l9Ck8yR3X2xm95vZqFoOfxJoCSwiFDjPu3secCpwNaEQyQl/nR8+5nfA\nCDNbCYwIb0e9gi27GftCNl1aN+fpq4fSLCE+6JJERA4Lc/faW0W5rKwsz87ODuz1d5SUc+lT/6Fo\n2x7e/q9TOLpTq8BqERGpKzOb5+5ZtbXThf2HqKKyiptfXUD+xp1M/P4whYSINDpawuMQ/fqdpcxc\nvpH7Rh3Ht/pooT8RaXwUFIfgxdlf8Pwnq7n+1F5876SeQZcjInJEKCgO0qwVG/nltMWc1a8TP79A\nC/2JSOOloDgIKzfs4Mcvz6dPp5Y8cWWmFvoTkUZNQVFPX+4s5fqJc2mWGM+z12bRUgv9iUgjp6Co\nh9KKSn744jyKt5fyf9cMpVu7lKBLEhE54vTrcB25O/e8uZDsL7bwx6syyewR07fJEBGpM/Uo6ujJ\nmfm8taCQ20f05cJBaUGXIyLSYBQUdfC3vHU8/I8VjBmSxk/OOjrockREGpSCohY5a7dy+6QchvZs\nx+8uGaSF/kSkyVFQ1KBw6x5+MDGb1FbN+PPVQ2meqIX+RKTp0WT2AewsreCGv8yltLySV248kY4t\nmwVdkohIIBQU1aiscm55dQErNuzg+e8Po29nLfQnIk2XgqIav31nKR8sK+b+0cdxet/ous2qiEhD\n0xzFfl6Zs4ZnP/6c607J4JqTM4IuR0QkcAqKCJ/kb+LeqYs4vW8qv9BCfyIigIJin/zinYx7aR5H\npbbgD1dlkhCvvxoREVBQALB5Vxk3TJxLUnwcE649gdbNE4MuSUQkajT5yezSikrGvTiPddtKePXG\nE+neXgv9iYhEatI9CnfnZ28t4tPVm/n9pYMY2rN90CWJiESdJh0UT/3rM96cX8AtZ/dh9JD0oMsR\nEYlKTToourdL4bKh3bj1nD5BlyIiErWa9BzFRYPTuGiwlgwXEalJk+5RiIhI7RQUIiJSIwWFiIjU\nSEEhIiI1UlCIiEiNFBQiIlIjBYWIiNRIQSEiIjUydw+6hkNmZhuBLw7y8I7ApsNYzuGiuupHddWP\n6qqfaK0LDq22nu5e6208G0VQHAozy3b3rKDr2J/qqh/VVT+qq36itS5omNo09CQiIjVSUIiISI0U\nFPBM0AUcgOqqH9VVP6qrfqK1LmiA2pr8HIWIiNRMPQoREalRkw0KM3vOzIrNbFHQtUQys+5mNtPM\nlprZYjO7JeiaAMysuZl9ama54bruC7qmSGYWb2YLzGx60LXsZWarzWyhmeWYWXbQ9exlZm3NbLKZ\nLQv/nJ0cBTUdE/572vu13cxuDbouADO7Lfwzv8jMXjWz5kHXBGBmt4RrWnyk/66a7NCTmQ0HdgIv\nuPuAoOvZy8y6Al3dfb6ZtQLmAWPcfUnAdRnQwt13mlki8DFwi7vPDrKuvczsdiALaO3uFwZdD4SC\nAshy96i6/t7MJgIfufuzZpYEpLj71qDr2svM4oFC4ER3P9jPRx2uWtIJ/az3d/c9ZjYJeMfd/xJw\nXQOA14BhQBkwA/iRu688Eq/XZHsU7j4L2Bx0Hftz93XuPj/8/Q5gKRD4Db09ZGd4MzH8FRW/ZZhZ\nN+AC4Nmga4l2ZtYaGA5MAHD3smgKibCzgc+CDokICUCymSUAKUBRwPUAHAvMdvfd7l4B/Au4+Ei9\nWJMNilhgZhlAJjAn2EpCwsM7OUAx8J67R0VdwOPAnUBV0IXsx4F/mNk8MxsbdDFhRwEbgefDQ3XP\nmlmLoIvazxXAq0EXAeDuhcDDwBpgHbDN3f8RbFUALAKGm1kHM0sBzge6H6kXU1BEKTNrCbwJ3Oru\n24OuB8DdK919CNANGBbu/gbKzC4Eit19XtC1VONUdz8eOA/4cXi4M2gJwPHAU+6eCewC7g62pK+E\nh8JGAW8EXQuAmbUDRgO9gDSghZl9L9iqwN2XAg8C7xEadsoFKo7U6ykoolB4DuBN4GV3fyvoevYX\nHqr4EBgZcCkApwKjwvMBrwFnmdlLwZYU4u5F4T+LgbcJjScHrQAoiOgNTiYUHNHiPGC+u28IupCw\nc4DP3X2ju5cDbwGnBFwTAO4+wd2Pd/fhhIbRj8j8BCgook540ngCsNTdHw26nr3MLNXM2oa/Tyb0\nH2hZsFWBu9/j7t3cPYPQkMU/3T3w3/jMrEX4YgTCQzvfJjRcECh3Xw+sNbNjwrvOBgK9UGI/VxIl\nw05ha4CTzCwl/H/zbELzhoEzs07hP3sA3+EI/r0lHKknjnZm9ipwBtDRzAqA/3H3CcFWBYR+Q74a\nWBieDwD4mbu/E2BNAF2BieErUuKASe4eNZeiRqHOwNuh9xYSgFfcfUawJe3zE+Dl8DDPKuD7AdcD\nQHisfQTww6Br2cvd55jZZGA+oaGdBUTPp7TfNLMOQDnwY3ffcqReqMleHisiInWjoScREamRgkJE\nRGqkoBARkRopKEREpEYKChERqZGCQkREaqSgEBGRGikoRESkRv8fMHgXl0o1jB8AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ad9aad0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### min_child_samples=10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 列采样参数 sub_feature/feature_fraction/colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n",
      "[CV] colsample_bytree=0.5 ............................................\n",
      "[CV] colsample_bytree=0.5 ............................................\n",
      "[CV] colsample_bytree=0.5 ............................................\n",
      "[CV] colsample_bytree=0.6 ............................................\n",
      "[CV] ...... colsample_bytree=0.5, score=-0.486140075251, total= 1.5min\n",
      "[CV] colsample_bytree=0.6 ............................................\n",
      "[CV] ...... colsample_bytree=0.5, score=-0.473416221102, total= 1.6min\n",
      "[CV] colsample_bytree=0.6 ............................................\n",
      "[CV] ...... colsample_bytree=0.6, score=-0.470299489856, total= 1.6min\n",
      "[CV] colsample_bytree=0.7 ............................................\n",
      "[CV] ...... colsample_bytree=0.5, score=-0.485259126755, total= 1.6min\n",
      "[CV] colsample_bytree=0.7 ............................................\n",
      "[CV] ...... colsample_bytree=0.6, score=-0.487786670046, total= 1.8min\n",
      "[CV] colsample_bytree=0.7 ............................................\n",
      "[CV] ...... colsample_bytree=0.7, score=-0.472198609029, total= 1.8min\n",
      "[CV] ...... colsample_bytree=0.6, score=-0.490648242826, total= 1.8min\n",
      "[CV] colsample_bytree=0.8 ............................................\n",
      "[CV] colsample_bytree=0.8 ............................................\n",
      "[CV] ...... colsample_bytree=0.7, score=-0.484990829732, total= 1.8min\n",
      "[CV] colsample_bytree=0.8 ............................................\n",
      "[CV] ...... colsample_bytree=0.7, score=-0.487944149665, total= 1.6min\n",
      "[CV] colsample_bytree=0.9 ............................................\n",
      "[CV] ...... colsample_bytree=0.8, score=-0.489349976332, total= 1.7min\n",
      "[CV] colsample_bytree=0.9 ............................................\n",
      "[CV] ...... colsample_bytree=0.8, score=-0.476967525175, total= 1.7min\n",
      "[CV] ...... colsample_bytree=0.8, score=-0.485731809243, total= 1.7min\n",
      "[CV] colsample_bytree=0.9 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  7.0min remaining:  1.8min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ....... colsample_bytree=0.9, score=-0.47507190897, total= 1.1min\n",
      "[CV] ...... colsample_bytree=0.9, score=-0.485958789103, total= 1.1min\n",
      "[CV] ...... colsample_bytree=0.9, score=-0.493741607327, total= 1.1min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:  8.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', colsample_bytree=1.0, learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=10, min_child_weight=5,\n",
       "        min_split_gain=0.0, n_estimators=374, n_jobs=4, num_class=9,\n",
       "        num_leaves=70, objective='multiclass', random_state=0,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=50000, subsample_freq=1),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':10,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48160472978\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8VvX5//HXlUEIIcyEGTDsISsQ\nllaruABbcIAMURwI1lq1YhVq+2urtSqlxdqissUFIrVKlaEiOJBEgmySkDDDTCAQSCD7+v1x3/C9\npSG5E5KcO8n1fDzyyH3O/Tn3uc4h5J0zPp8jqooxxhhzKX5OF2CMMca3WVAYY4wplgWFMcaYYllQ\nGGOMKZYFhTHGmGJZUBhjjCmWBYUxxphiWVAYY4wplgWFMcaYYgU4XUB5CAsL08jISKfLMMaYKmXj\nxo3HVTW8pHbVIigiIyOJi4tzugxjjKlSRGS/N+28OvUkIoNFJFFEkkVkSjHtRoiIiki0ezpQRBaK\nyDYRiReRqR5t97nnbxaROI/5jUTkcxFJcn9v6E2NxhhjKkaJQSEi/sBMYAjQFRgjIl2LaBcKPAbE\nesweCQSpanegDzBJRCI93r9eVXuparTHvCnAalXtAKx2TxtjjHGIN0cU/YBkVd2jqrnAYmB4Ee2e\nB6YB2R7zFAgRkQAgGMgFTpewvuHAQvfrhcBtXtRojDGmgngTFC2BFI/pg+55F4hIFNBKVT+5aNml\nQBZwBDgATFfVdPd7CnwmIhtFZKLHMk1V9QiA+3sTbzfGGGNM+fPmYrYUMe/CQyxExA+YAdxXRLt+\nQAHQAmgIfCMiX6jqHuBqVT0sIk2Az0UkQVW/9rZwd7hMBGjdurW3ixljjCklb44oDgKtPKYjgMMe\n06FAN2CtiOwDBgDL3Be0xwIrVTVPVVOBdUA0gKoedn9PBf6DK1QAjolIcwD399SiilLV2aoararR\n4eEl3t1ljDGmjLwJig1ABxFpIyK1gNHAsvNvqmqGqoapaqSqRgIxwDBVjcN1ummQuITgCpEEEQlx\nX/zGPf9mYLv7I5cB492vxwMfX/ZWGmOMKbMSg0JV84FHgVVAPLBEVXeIyHMiMqyExWcCdXGFwAZg\ngapuBZoC34rIFuB74FNVXele5iXgJhFJAm5yTxtjjPGQcTaPPy7bwZnsvApfl1SHZ2ZHR0erdbgz\nxtQUMXtO8OT7m0k9k8Mb4/pwY9emZfocEdl4UfeEIlWLntnGGFMT5OYXMuOLXbzx1W4iG4fw719c\nRc9WDSp8vRYUxhhTBexOy+SJxZvZdiiD0X1b8fufdSUkqHJ+hVtQGGOMD1NVFm9I4bn/7iQo0I83\nxvVmcLfmlVqDBYUxxvio9Kxcpvx7K5/tPMbV7Rvzt5G9aFa/dqXXYUFhjDE+6JukNCYv2cKps3k8\nO7QLD/6kDX5+RfV/rngWFMYY40Oy8wr466pE5n27l/ZN6rLg/r5c2aK+ozVZUBhjjI/YdewMjy3a\nRMLRM9w78AqmDulCcC1/p8uyoDDGGKepKm+t389flscTWjuA+fdFM6hz2fpGVAQLCmOMcVDamRye\nXrqFNYlpXN8pnGkjehIeGuR0WT9iQWGMMQ75MuEYv/lgK5k5+Tw3/EruGXAFIs5csC6OBYUxxlSy\nc7kF/GV5PG/H7Kdzs1AWTRxAx6ahTpd1SRYUxhhTiXYczuDxxZtJTs1kwk/a8JvBnQgKcP6CdXEs\nKIwxphIUFirzvt3LtFUJNKxTi7cf7Mc1HarGs3QsKIwxpoIdzchm8gebWZd8gluubMqLd/SgUUgt\np8vymgWFMcZUoJXbjzDlw23k5BXy0h3dGdW3lU9esC6OBYUxxlSArJx8nvvvTt6PS6F7y/r8Y3Qv\n2obXdbqsMrGgMMaYcrYl5RSPL97E/vSzPHJdO564sSO1Arx58rRvsqAwxphyUlCovPHVbmZ8vosm\noUEsemgAA9o2drqsy2ZBYYwx5eDgybM8+f4Wvt+Xzs96NOeF27pTv06g02WVCwsKY4y5TB9vPsTv\nPtqOKvz9rp7cHtWyyl2wLo4FhTHGlNHp7Dz+8PEO/rPpEL1bN+CVUVG0blzH6bLKnVdXV0RksIgk\nikiyiEwppt0IEVERiXZPB4rIQhHZJiLxIjL1ovb+IrJJRD7xmPemiOwVkc3ur15l3ThjjKkocfvS\nGfqPb1i25TBP3NiBJZMGVsuQAC+OKETEH5gJ3AQcBDaIyDJV3XlRu1DgMSDWY/ZIIEhVu4tIHWCn\niCxS1X3u9x8H4oF6F632N6q6tCwbZIwxFSm/oJBXv0zmX18m0bJhMEsmDaTPFQ2dLqtCeXNE0Q9I\nVtU9qpoLLAaGF9HueWAakO0xT4EQEQkAgoFc4DSAiEQAtwJzy16+McZUnv0nshjxxnpeXZ3E7VER\nLH/smmofEuBdULQEUjymD7rnXSAiUUArVf2EH1sKZAFHgAPAdFVNd7/3CvA0UFjEOl8Qka0iMkNE\nihyYXUQmikiciMSlpaV5sRnGGFM2qsoHcSkM/cc37EnL5J9jovjbXT0JrV097moqiTdBUdSle73w\npogfMAOYXES7fkAB0AJoA0wWkbYi8jMgVVU3FrHMVKAz0BdoBDxTVFGqOltVo1U1Ojy8agysZYyp\nejLO5vHoe5v4zdKtdGtZnxVPXMvPe7ZwuqxK5c1dTweBVh7TEcBhj+lQoBuw1n07WDNgmYgMA8YC\nK1U1D0gVkXVANBAFDBORoUBtoJ6IvKOq41T1iPtzc0RkAfBU2TfPGGPK7rvdx5m8ZIvrKXSDOzHp\n2nb4+1Wf21695c0RxQagg4i0EZFawGhg2fk3VTVDVcNUNVJVI4EYYJiqxuE63TRIXEKAAUCCqk5V\n1Qh3+9HAl6o6DkBEmru/C3AbsL28NtYYY7yRm1/ISysSuHtuLMGB/vznkat55Lr2NTIkwIsjClXN\nF5FHgVWAPzBfVXeIyHNAnKouK2bxmcACXL/sBVigqltLWOW7IhLubr8ZeNiL7TDGmHKxOy2Txxdv\nYvuh04zp15rf/6wLdWrV7C5noqolt/Jx0dHRGhcX53QZxpgqTFV57/sDPP/JToID/Xn5zh7cfGUz\np8uqUCKyUVWjS2pXs2PSGGOAE5k5PPPvbXwRf4xrOoQxfWRPmtar7XRZPsOCwhhTo321K42nPthC\nxtk8fv+zrtx/VSR+NfRaxKVYUBhjaqTsvAKmrUxk/rq9dGxal7ce6EeX5hcPEmHAgsIYUwMlHj3D\n44s3kXD0DPddFcmUIZ2pHejvdFk+y4LCGFNjqCpvfrePF1ckUK92IAvu78v1nZo4XZbPs6AwxtQI\nqWey+c0HW/lqVxo3dG7CyyN6EFa3yBGCzEUsKIwx1d4XO4/x9L+3kpWTz/O3dWNc/9bV6sFCFc2C\nwhhTbZ3LLeCF5Tt5J+YAXZvX49UxvWjfJNTpsqocCwpjTLW0/VAGjy/exO60LCZe25bJN3ckKMAu\nWJeFBYUxplopLFTmfLOH6Z8l0iikFu9O6M/V7cOcLqtKs6AwxlQbRzLOMXnJFr7bfYIh3Zrxl9u7\n0zCkltNlVXkWFMaYamHFtiNM+XAbeQWFTLuzByOjI+yCdTmxoDDGVGlZOfn86b87WBJ3kJ4R9Xll\ndBRtwkKcLqtasaAwxlRZmw6c5In3N5OSfpZHr2/P4zd2INDfm8fsmNKwoDDGVDkFhcpra5J5ZXUS\nzerVZvHEgfRr08jpsqotCwpjTJWSkn6WJ5dsZsO+kwzv1YLnhnejfnCg02VVaxYUxpgq4+PNh/jd\nf1xPR35lVC9ui2rpcEU1gwWFMcbnnc7O4/cfbefjzYeJvqIhM0b1olWjOk6XVWNYUBhjfNqGfek8\nsXgzR09n8+RNHXnkunYE2AXrSmVBYYzxSXkFhby6OomZa5Jp1agOSx8eSFTrhk6XVSNZUBhjfM6+\n41k8/v5mtqScYmSfCP4w7ErqBtmvK6d4dfwmIoNFJFFEkkVkSjHtRoiIiki0ezpQRBaKyDYRiReR\nqRe19xeRTSLyice8NiISKyJJIvK+iFj/e2NqCFVlSVwKQ1/9hn3Hs3jt7t78dWRPCwmHlRgUIuIP\nzASGAF2BMSLStYh2ocBjQKzH7JFAkKp2B/oAk0Qk0uP9x4H4iz7qZWCGqnYATgIPersxxpiq62xu\nPr987weeXrqVnhENWPnENQzt3tzpsgzeHVH0A5JVdY+q5gKLgeFFtHsemAZke8xTIEREAoBgIBc4\nDSAiEcCtwNzzjcU1MMsgYKl71kLgttJskDGmanp1dTIrth9lypDOvDuhP83rBztdknHzJihaAike\n0wfd8y4QkSiglap+wo8tBbKAI8ABYLqqprvfewV4Gij0aN8YOKWq+Zdal8c6J4pInIjEpaWlebEZ\nxhhfdfDkWeav28vtUS15+Kft8POzwfx8iTdBUdS/mF54U8QPmAFMLqJdP6AAaAG0ASaLSFsR+RmQ\nqqobS7OuH81Una2q0aoaHR4e7sVmGGN81fRViQjw1M2dnC7FFMGbK0QHgVYe0xHAYY/pUKAbsNY9\npG8zYJmIDAPGAitVNQ9IFZF1QDQQBQwTkaFAbaCeiLwD3AM0EJEA91HFxesyxlQzWw+e4qPNh/nl\n9e1o0cBON/kib44oNgAd3Hcj1QJGA8vOv6mqGaoapqqRqhoJxADDVDUO1+mmQeISAgwAElR1qqpG\nuNuPBr5U1XGqqsAaYIT748cDH5fPphpjfI2q8sKn8TQOqcXDP23ndDnmEkoMCvdf9o8Cq3DdobRE\nVXeIyHPuo4bizATqAttxBc4CVd1awjLPAE+KSDKuaxbzSqrRGFM1fRGfSuzedJ64qSOhtW1gP18l\nrj/iq7bo6GiNi4tzugxjTCnkFRRyyytfA7DqiWvtORIOEJGNqhpdUjv7lzHGOGLx9wfYk5bF1CFd\nLCR8nP3rGGMq3ZnsPF75Ion+bRpxY5cmTpdjSmD94o0xle6Nr3ZzIiuXBbd2wX23pPFhdkRhjKlU\nh0+dY+43e7mtVwt6RDRwuhzjBQsKY0ylmv5ZIgo8dYt1rqsqLCiMMZVm+6EM/rPpEA9c3YaIhvaE\nuqrCgsIYUynOd65rEBzII9db57qqxILCGFMp1iSmsn7PCR6/oQP1rHNdlWJBYYypcPkFhby4PIE2\nYSGM7X+F0+WYUrKgMMZUuCVxB0lKzeSZwZ2pFWC/dqoa+xczxlSozJx8/v75LvpGNuSWK5s6XY4p\nAwsKY0yFmv3Vbo5n5vDboda5rqqyoDDGVJijGdnM/mYPP+/ZgqjWDZ0ux5SRBYUxpsL87bNECgvh\naetcV6VZUBhjKsTOw6dZ+sNBxl91Ba0aWee6qsyCwhhTIV5cEU+92oE8en0Hp0sxl8mCwhhT7r7a\nlcY3Scd57IYO1K9jneuqOgsKY0y5KihU/vJpPFc0rsM9A6xzXXVgQWGMKVdLN6aQeOyMda6rRuxf\n0RhTbrJy8vnbZ7vo3boBQ7o1c7ocU068CgoRGSwiiSKSLCJTimk3QkRURKLd04EislBEtolIvIhM\ndc+vLSLfi8gWEdkhIn/y+Iw3RWSviGx2f/W63I00xlSOOd/sIfVMDs/ak+uqlRIfhSoi/sBM4Cbg\nILBBRJap6s6L2oUCjwGxHrNHAkGq2l1E6gA7RWQRsB8YpKqZIhIIfCsiK1Q1xr3cb1R16WVvnTGm\n0qSezmbWV3sY2r0Zfa5o5HQ5phx5c0TRD0hW1T2qmgssBoYX0e55YBqQ7TFPgRARCQCCgVzgtLpk\nutsEur+0jNtgjPEBf/98F/mFhTx9S2enSzHlzJugaAmkeEwfdM+7QESigFaq+slFyy4FsoAjwAFg\nuqqmu5fxF5HNQCrwuap6Hom8ICJbRWSGiASVaouMMZUu8egZlsSlcM+ASCLDQpwux5Qzb4KiqBON\nF/76FxE/YAYwuYh2/YACoAXQBpgsIm0BVLVAVXsBEUA/EenmXmYq0BnoCzQCnimyKJGJIhInInFp\naWlebIYxpqK8uCKeukEB/GpQe6dLMRXAm6A4CLTymI4ADntMhwLdgLUisg8YACxzX9AeC6xU1TxV\nTQXWAdGeH66qp4C1wGD39BH3qakcYAGusPkfqjpbVaNVNTo8PNyLzTDGVIRvktJYm5jGrwZ1oGFI\nLafLMRXAm6DYAHQQkTYiUgsYDSw7/6aqZqhqmKpGqmokEAMMU9U4XKebBolLCK4QSRCRcBFpACAi\nwcCNQIJ7urn7uwC3AdvLaVuNMeWsoND1HOyIhsHce5V1rquuSrzrSVXzReRRYBXgD8xX1R0i8hwQ\np6rLill8Jq6jgu24TmEtUNWtItIDWOi+o8oPWOJxfeNdEQl3t98MPFzWjTPGVKwPfzhIwtEzvDom\niqAAf6fLMRVEVKv+zUbR0dEaFxfndBnG1Cjncgu4bvoamtUP5qNHrrJ+E1WQiGxU1eiS2lnPbGNM\nmcz9Zg/HTufwrD25rtqzoDDGlFramRze+Go3t1zZlH5trHNddWdBYYwptVe+2EVOfiHPDLbOdTWB\nBYUxplSSjp1h8YYUxg24grbhdZ0ux1QCCwpjTKm8tCKBOoH+PHaDPbmuprCgMMZ47bvk46xOSOWR\n69vTyDrX1RgWFMYYrxQWKi8sj6dlg2DuvzrS6XJMJbKgMMZ45aPNh9hx+DS/uaUTtQOtc11NYkFh\njClRdl4B01cl0r1lfYb1bOF0OaaSWVAYY0o0f91eDmdk89uhXfDzs851NY0FhTGmWCcyc3htzW5u\n7NKUge0aO12OcYAFhTGmWP9YncS5vAKmDLHOdTWVBYUx5pJ2p2XybuwBxvRrRfsm1rmuprKgMMZc\n0ksrEggO9OeJGzs6XYpxkAWFMaZIMXtO8PnOY/ziunaE1bVH19dkFhTGmP9RWKj8ZXk8zevX5oGr\n2zhdjnGYBYUx5n/8d+thth7M4KmbOxFcyzrX1XQWFMaYH8nOK2DaykS6Nq/H7VEtnS7H+AALCmPM\njyz8bh+HTp3jd7da5zrjYkFhjLkgPSuXf61J5vpO4VzVPszpcoyPsKAwxlzw6uoksnLymTq0i9Ol\nGB/iVVCIyGARSRSRZBGZUky7ESKiIhLtng4UkYUisk1E4kVkqnt+bRH5XkS2iMgOEfmTx2e0EZFY\nEUkSkfdFxAa9N6YS7D2exTsx+xnVtzUdm4Y6XY7xISUGhYj4AzOBIUBXYIyIdC2iXSjwGBDrMXsk\nEKSq3YE+wCQRiQRygEGq2hPoBQwWkQHuZV4GZqhqB+Ak8GDZNs0YUxovr0igVoAfv77Jnlxnfsyb\nI4p+QLKq7lHVXGAxMLyIds8D04Bsj3kKhIhIABAM5AKn1SXT3SbQ/aUiIsAgYKn7vYXAbaXcJmNM\nKcXtS2fljqM8/NN2NAmt7XQ5xsd4ExQtgRSP6YPueReISBTQSlU/uWjZpUAWcAQ4AExX1XT3Mv4i\nshlIBT5X1VigMXBKVfMvtS5jTPlSVf78aTxN6wUx4RrrXGf+lzdBUdT9cXrhTRE/YAYwuYh2/YAC\noAXQBpgsIm0BVLVAVXsBEUA/EelW0rp+VJTIRBGJE5G4tLQ0LzbDGFOUT7cdYXPKKSbf3Ik6tQKc\nLsf4IG+C4iDQymM6AjjsMR0KdAPWisg+YACwzH1BeyywUlXzVDUVWAdEe364qp4C1gKDgeNAA/ep\nqqLW5bncbFWNVtXo8PBwLzbDGHOxnPwCXl6ZQOdmodzZO8LpcoyP8iYoNgAd3Hcj1QJGA8vOv6mq\nGaoapqqRqhoJxADDVDUO1+mmQeISgitEEkQkXEQaAIhIMHAjkKCqCqwBRrg/fjzwcblsqTHmf7y9\nfj8p6ef47dAu+FvnOnMJJQaF+3rBo8AqIB5Yoqo7ROQ5ERlWwuIzgbrAdlyBs0BVtwLNgTUistU9\n/3OP6xvPAE+KSDKuaxbzyrBdxpgSnDqby6urk7i2YzjXdrSjcnNpXp2QVNXlwPKL5v2/S7S9zuN1\nJq5bZC9usxWIusTye3Bd2zA+ZnPKKb5KTOMX17WjVoD11azq/vllMpk5+fx2qD25zhTPrlwZr/xw\n4CT3zI0lK7eAHw6c5I1xfWxU0SrswImzvLV+HyP7tKJzs3pOl2N8nP1ZaEq09eApxs/7nrDQIKYO\n6cw3SWncOz+W09l5TpdmyujlVQkE+Pnx5M325DpTMgsKU6zthzIYNzeWBiGBLHpoAJN+2o5/junN\n5pRTjJkdw4nMHKdLNKW0cf9JPt16hInXtqVpPetcZ0pmQWEuKeHoae6ZF0to7UDemzCAFg2CAbi1\nR3Pm3BvN7rRM7pq1niMZ5xyu1HhL1fXkuvDQICZe29bpckwVYUFhipR07Ax3z4klKMCf9x7qT6tG\ndX70/nWdmvDWA/1JPZ3DiNfXs+94lkOVmtJYuf0oG/ef5MmbOhISZJcojXcsKMz/2J2WyZg5sfj5\nCe891J8rGocU2a5fm0YsmjiAc3kFjHhjPQlHT1dypaY0cvMLeWllAh2b1mVkH+tcZ7xnQWF+ZN/x\nLMbOiQGURQ/1p2143WLbd2tZnyWTBhDgJ4yaFcOmAycrp1BTau/E7Gf/ibNMHdqFAH/7r2+8Zz8t\n5oKU9LOMnRNDbn4h704YQPsm3j2ToH2TUD54eCAN6gRy99xYvks+XsGVmtLKOJfHq18m8ZP2YVxn\nnetMKVlQGAAOnTrHmDkxZOUW8M6E/nRqVroH17RqVIcPJg2kVcM63PfmBj7feayCKjVl8dqaZDLO\n5TF1aGdco/kb4z0LCsPRjGzGzI4h41we7zzYnytb1C/T5zSpV5v3Jw2gS/N6PPzORj7adKicKzVl\nkZJ+lgXr9nFn74gy/9uams2CooZLPZ3N2DkxpGfl8tYD/egecXm/SBrUqcW7E/rTL7IRv16ymbdj\n9pdTpaas/roqET8/mGyd60wZWVDUYGlnchg7N5ajp7N58/6+RLVuWC6fWzcogAX39+WGzk34/Ufb\neW1tcrl8rim9zSmnWLblMBN+0pbm9YOdLsdUURYUNVR6Vi7j5sZy8ORZFtzXl+jIRuX6+bUD/Xl9\nXB+G92rBtJWJvLQiAdco8qayqCp/+TSesLq1ePi6dk6XY6ow63FTA5066wqJfSeyWHBfX/q3bVwh\n6wn092PGXb2oGxTAG1/t5kx2Hs8P74afPfegUny28xjf70vnz7d1o651rjOXwX56apiMc3ncM+97\nklMzmTs+mqvah1Xo+vz8hD/f1o3Q2oG88dVuMnPymT6yJ4F2H3+Fyiso5KUVCbQLD2F031YlL2BM\nMSwoapAz2XmMn/89CUdPM+uePpX2sBoRYcqQztQLDmDaykSycgr419goagfaMOUVZdH3B9h7PIt5\n46Otc525bPYTVENk5eRz/4INbD+UwcyxvRnUuWml1/DIde15fviVfBF/jPsXbCAzJ7/Sa6gJTmfn\n8coXSQxs25hBnZs4XY6pBiwoaoCzufnc/+YGNqWc4p9jorj5ymaO1XLPwEhmjOrJ9/vSGTc3llNn\ncx2rpbp6fe1u0rNyefbWLta5zpQLC4pqLjuvgAkL44jbl86MUb0Y0r250yVxe1QEr9/dm52HTzNq\nVgypp7OdLqnaOHTqHPO+3cvtUS3p1tI615nyYUFRjWXnFfDQW3Gs33OC6SN7MqxnC6dLuuDmK5ux\n4P6+pJw8y8hZ60lJP+t0SdXC9FWJADx1SyeHKzHViQVFNZWbX8gj7/7AN0nHefmOHtzR2/eGlb66\nfRjvTOjPyaxcRr6xnuTUTKdLqtK2HczgP5sO8eBP2tCygXWuM+XHq6AQkcEikigiySIypZh2I0RE\nRSTaPR0oIgtFZJuIxIvIVPf8ViKyxj1vh4g87vEZfxSRQyKy2f019HI3sqbJKyjk0fd+4MuEVF64\nvRt3+fDtkb1bN+T9SQPJL1TumrWe7YcynC6pSlJVXli+k0YhtfiFda4z5azEoBARf2AmMAToCowR\nka5FtAsFHgNiPWaPBIJUtTvQB5gkIpFAPjBZVbsAA4BfXvSZM1S1l/treZm2rIbKLyjk8cWb+Gzn\nMf407Eru7n+F0yWVqEvzenzw8ECCA/0ZMzuGDfvSnS6pylkdn0rMnnSeuLED9WoHOl2OqWa8OaLo\nBySr6h5VzQUWA8OLaPc8MA3wvDKpQIiIBADBQC5wWlWPqOoPAKp6BogHWpZ9MwxAQaHy5JItLN92\nlN/d2oXxV0U6XZLX2oSF8MHDAwmvF8Q982JZm5jqdElVRn5BIS+uiKdtWAhj+rV2uhxTDXkTFC2B\nFI/pg1z0S11EooBWqvrJRcsuBbKAI8ABYLqqpl+0bCQQxY+PRB4Vka0iMl9EymekumquoFD5zdIt\nLNtymClDOjPhmrZOl1RqLRoEs2TSQNqG1eWht+JYvu2I0yVVCYs3pLA7LYspQzpbj3dTIbz5qSrq\nRuwLo7uJiB8wA5hcRLt+QAHQAmgDTBaRth7L1gX+DTyhqucfuPw60A7ohStg/lZkUSITRSROROLS\n0tK82Izqq7BQ+e2H2/jwh0NMvqkjD/+06p6jDqsbxKKJA+gZ0YBH3/uBJRtSSl6oBjuTnccrX+yi\nX2Qjbupa+Z0oTc3gTVAcBDyvhkYAhz2mQ4FuwFoR2YfrmsMy9wXtscBKVc1T1VRgHXDhQjeukHhX\nVT88/2GqekxVC1S1EJiDK2z+h6rOVtVoVY0OD6+5j3ZUVX7/8Xbej0vhsUHt+dUNHZwu6bLVDw7k\nrQf7cXX7MJ7+91bmfbvX6ZJ81qyv9nA8M5ffWuc6U4G8CYoNQAcRaSMitYDRwLLzb6pqhqqGqWqk\nqkYCMcAwVY3DdbppkLiE4AqRBHH9RM8D4lX1754rExHPHmG3A9svY/uqNVXlT//dybuxB/jFde34\n9U3V58E0dWoFMHd8NEO6NeP5T3byyhe7bJjyixzJOMecb/YwrGcLerVq4HQ5phorMShUNR94FFiF\n66LzElXdISLPiciwEhafCdTF9ct+A7BAVbcCVwP34AqRi2+Dnea+nXYrcD3w6zJtWTWnqrzwaTxv\nfrePCT9pw9O3dKp2f1EGBfjzzzFRjOgTwStfJPH8J/EWFh6mr9qFKvzGOteZCubV6LHuW1SXXzTv\n/12i7XUerzNx3SJ7cZtvKfqyDbeXAAATuklEQVTaB6p6jzc11WSqyrRVicz9di/3XRVZrcf0CfD3\nY9qdPQitHcD8dXvJzMnjxTt64F/Dn2mx43AGH246yMRr2tKqUR2nyzHVnA0zXgXN+CKJ19fu5u7+\nrfnDz7tW25A4z89P+H8/60q92oH8Y3USmTn5vDIqiloBNfMOH1XlL8vjqR8cyCPXt3e6HFMDWFBU\nMf9cncSrq5MYFd2K54d3q/YhcZ6I8OubOhJaO4A/fxpPVk4cb4zrQ3CtmvdMi7W70liXfII//Lwr\n9YOtc52peDXzT7Iq6o2vdvO3z3dxR++WvHhH9xr5SNEJ17Tl5Tu7801SGvfOj+V0dp7TJVWq/IJC\n/vJpPJGN61SJXvemerCgqCLmfrOHl1YkMKxnC/46omeNDInzRvVtzT/H9GZzyinGzI7hRGaO0yVV\nmg82HiQpNZNnBneusafeTOWzn7QqYOF3+/jzp/EM7d6Mv9/Vs8ZfyAW4tUdzZt8bze60TO6atZ4j\nGeecLqnCZeXk87fPdtHnioYM7ubcw6dMzWNB4ePejd3PH5bt4KauTfnH6Ch7/rGH6zs14a0H+nPs\ndA4jXl/PvuNZTpdUoWZ9vYfjmTnV+i4345vst44PW7IhhWf/s51BnZvwr7FRNo5PEfq1acSihwZw\nLq+AkbPWk3D0dMkLVUHHTmcz5+s93NqjOb1b2/BnpnLZbx4f9eEPB3nmw61c2zGc1+7uTVBAzbu7\nx1vdI+qzZNIA/ARGzYph04GTTpdU7v7+2S7yCwt55pbOTpdiaiALCh+0bMthnvpgC1e1a8zse/pQ\nO9BCoiTtm4Sy9OGraFAnkLvnxvLd7uNOl1Ru4o+cZsnGFMYPjKR1Y+tcZyqfBYWPWb7tCL9+fzN9\nIxsx996+FhKl0KpRHT6YNJCIhsHct2ADn+885nRJ5eLFFQmEBgXw6CDrXGecYUHhQz7bcZTHFm0i\nqlUD5t/Xt0Z2JrtcTerV5v2JA+nSvB4Pv7ORjzcfcrqky/LVrjS+3pXGYzd0oEGdWk6XY2ooCwof\nsSYhlV++9wPdWtZnwf19CQmyTvNl1TCkFu9O6E/fyIY88f5m3o7Z73RJZVJQqLy4PJ5WjYK5Z6B1\nrjPOsaDwAV/vSmPSOxvp3KweCx/oR6g98/iy1Q0K4M37+3FD5yb8/qPtvLY22emSSu3fGw+ScPQM\nzwzubDczGEdZUDjsu+TjPPRWHO3D6/L2g/1s7J5yVDvQn9fH9WFYzxZMW5nIyysTqsww5Wdz85n+\nWSK9WjXg1u7NS17AmApk5zccFLvnBA8ujCOycQjvTOhv56ArQKC/HzNG9SK0dgCvr93Nmew8nhvW\nzeeHQJnz9V5Sz+Tw2t29rXOdcZwFhUM27k/n/jc30LJhMO8+1J9GIRYSFcXfT/jzbd2oWzuAWV/t\nITM7n7+O7OmzHRhTz2Qz6+vdDOnWjOjIRk6XY4wFhRM2p5xi/PwNNKtXm/cm9CesbpDTJVV7IsLU\nIV2oHxzItJWJZOYU8K+xUT55+/GMz5PIzS/k6cHWuc74Bt/8k6oa23Ywg3vmxdK4bi3ee2gATerV\ndrqkGuWR69rz/PAr+SL+GA+8uYHMnHynS/qRXcfO8P6GA4wbcAVtwkKcLscYwIKiUu08fJpx82Kp\nHxzIew8NoFl9Cwkn3DMwkhmjehK7N51xc2M5dTbX6ZIueHF5PCFBATx2QwenSzHmAguKSpJ49Azj\n5sUSUsufRQ8NoGWDYKdLqtFuj4rg9bt7s/PwaUbPjiH1TLbTJfFt0nHWJKbx6PXt7ZqV8SkWFJUg\nOfUMd8+NIdBfeO+hAbRqZOP1+IKbr2zGgvv7ciD9LCPfWE9K+lnHaikoVF5YHk/LBsGMvyrSsTqM\nKYpXQSEig0UkUUSSRWRKMe1GiIiKSLR7OlBEForINhGJF5Gp7vmtRGSNe94OEXnc4zMaicjnIpLk\n/l6lx1Tek5bJmDmxiLhCItLOO/uUq9uH8c6E/pzMyuWuWetJTs10pI7/bDpE/JHTPD24k09eYDc1\nW4lBISL+wExgCNAVGCMiXYtoFwo8BsR6zB4JBKlqd6APMElEIoF8YLKqdgEGAL/0+MwpwGpV7QCs\ndk9XSftPZDF2TiyFhcp7E/rTLryu0yWZIvRu3ZD3Jw0kr0C5a9Z6th/KqNT1n8st4G+fJdIzoj4/\n79GiUtdtjDe8OaLoBySr6h5VzQUWA8OLaPc8MA3wPNmrQIiIBADBQC5wWlWPqOoPAKp6BogHWrqX\nGQ4sdL9eCNxWuk3yDSnpZxk7J5ac/ALefag/HZqGOl2SKUaX5vX44OGBBAf6M2Z2DBv2pVfauuev\n28uRjGx+O7SLz3cENDWTN0HREkjxmD7I//1SB0BEooBWqvrJRcsuBbKAI8ABYLqqpl+0bCQQxf8d\niTRV1SMA7u9NiipKRCaKSJyIxKWlpXmxGZXn8KlzjJ0bw5nsPN5+sD+dm9VzuiTjhTZhIXzw8EDC\n6wVxz7xYvtpV8T9XaWdyeG1NMjd1bUr/to0rfH3GlIU3QVHUnzgXBswRET9gBjC5iHb9gAKgBdAG\nmCwibT2WrQv8G3hCVUv1DEtVna2q0aoaHR4eXppFK9TRjGzGzonhVFYe70zoT7eW9Z0uyZRCiwbB\nLJk0kLZhdZmwcAPLtx2p0PX9Y/UusvMLmTLEOtcZ3+VNUBwEWnlMRwCHPaZDgW7AWhHZh+uawzL3\nBe2xwEpVzVPVVGAdcOFCN66QeFdVP/T4vGMi0tzdpjmQWpYNc0LqmWzGzo3heGYuCx/sR4+IBk6X\nZMogrG4QiyYOoGdEAx597weWxKWUvFAZJKeeYdH3Kdzdv7VdvzI+zZug2AB0EJE2IlILGA0sO/+m\nqmaoapiqRqpqJBADDFPVOFynmwaJSwiuEEkQ1yhn84B4Vf37RetbBox3vx4PfHwZ21dpTmTmcPec\nWI5mZLPg/r70bl2lb9aq8eoHB/LWg/24un0YTy/dyvxv95b7Ol5akUBwoD+PW+c64+NKDApVzQce\nBVbhuui8RFV3iMhzIjKshMVnAnWB7bgCZ4GqbgWuBu7BFSKb3V9D3cu8BNwkIknATe5pn3YyK5e7\n58aScvIs88b3pa8N5FYt1KkVwNzx0Qzp1oznPtnJK1/sKrdhytfvPsEX8ak8cn07GttYX8bHSVUZ\nn7840dHRGhcX58i6M87mMXZuDMmpmcwb35efdAhzpA5TcfILCpny4TaWbjzIgz9pw+9u7XJZQ38X\nFirDZ67jRGYOXz51nfWbMI4RkY2qGl1SOxs99jKczs7j3vmxJB3LZPa9fSwkqqkAfz+m3dmDukEB\nzPt2L2ey83jxjh74l/FW1mVbDrPtUAZ/v6unhYSpEiwoyigzJ5/x879n55HTvDGuD9d1KvIuXlNN\n+PkJf/h5V+oFB/Lq6iSycgqYMaoXtQJKNwpOdl4Bf12VyJUt6nFbr5YlL2CMD7CgKIOzufncv+B7\nth3MYObdvbmhS1OnSzKVQER48qaO1KsdwJ8/jSczJ583xvUhuJb3RwUL1u3j0Klz/HVED+tcZ6oM\nGxSwlM7lFvDAmxvYuP8k/xgdxS1XNnO6JFPJJlzTlpfv7M7XSWncOz+W09l5Xi13ItPVue6Gzk24\nqr2dpjRVhwVFKWTnFfDQW3F8vzedGaN6cWsPe+h9TTWqb2v+OSaKzSmnGDsnhhOZOSUu8+rqJM7m\nFTB1qHWuM1WLBYWXcvILmPT2RtbtPs60ET0ZbueXa7yf9WjB7HujSTqWyV2z1nMk49wl2+5Oy+Td\n2AOM7tuK9k1s3C9TtVhQeCE3v5BfvvsDX+1K46U7ujOiT4TTJRkfcX2nJrz1QD+Onc5hxOvr2Xc8\nq8h2L69IICjAjydu7FjJFRpz+SwoSpBXUMivFv3AF/Gp/Pm2bozq29rpkoyP6d+2MYseGsDZ3HxG\nzlpPwtEfD1v2/d50Ptt5jId/2o7wUOtcZ6oeC4pi5BcU8sT7m1m14xh/+HlXxg24wumSjI/qHlGf\nJZMG4icwalYMm1NOAa7OdS98upOm9YKYcE3bEj7FGN9kQXEJBYXKUx9s4dOtR3h2aBfuv7qN0yUZ\nH9ehaShLH76K+sGB3D0nhu92H+eTbUfYcjCDp27uVKrbaI3xJdaPogiFhcoz/97KR5sP8/TgTjx0\nrf0laLzTqlEdPnh4IPfMi+W+BRuoVzuQLs3rcUdvu65lqi47orhIYaHy7EeucX1+fWNHHrmuvdMl\nmSqmab3avD9xIF2ahXI8M4ffDu1c5uE+jPEFdkThQVX5w7IdLPo+hV8Nas/jN9rwz6ZsGobUYtHE\nASQePUOUDTlvqjg7onBTVZ77ZCdvx+xn0k/b8uRNdhujuTx1agVYSJhqwYICV0i8uCKBBev28cDV\nbZgyuPNlDSNtjDHVSY0PClVl+meJzP56D/cOvILf/+zynjVgjDHVTY0Pin+sTmLmmt2M6deaP/78\nSgsJY4y5SI0OitfWJvPKF0mM7BPBC7d1s2GfjTGmCDU6KK5oFMLIPhG8dKc9G8AYYy6lRt8ee2uP\n5jZUuDHGlMCrIwoRGSwiiSKSLCJTimk3QkRURKLd04EislBEtolIvIhM9Wg7X0RSRWT7RZ/xRxE5\nJCKb3V9Dy7pxxhhjLl+JQSEi/sBMYAjQFRgjIl2LaBcKPAbEesweCQSpanegDzBJRCLd770JDL7E\nameoai/313LvNsUYY0xF8OaIoh+QrKp7VDUXWAwML6Ld88A0INtjngIhIhIABAO5wGkAVf0aSL+M\n2o0xxlQCb4KiJZDiMX3QPe8CEYkCWqnqJxctuxTIAo4AB4DpqupNODwqIlvdp6esa6sxxjjIm6Ao\n6nYgvfCmiB8wA5hcRLt+QAHQAmgDTBaRkoZifR1oB/TCFTB/K7IokYkiEicicWlpaSVuhDHGmLLx\nJigOAq08piOAwx7ToUA3YK2I7AMGAMvcF7THAitVNU9VU4F1QHRxK1PVY6paoKqFwBxcYVNUu9mq\nGq2q0eHh4V5shjHGmLLwJig2AB1EpI2I1AJGA8vOv6mqGaoapqqRqhoJxADDVDUO1+mmQeISgitE\nEopbmYh43q96O7D9Um2NMcZUvBKDQlXzgUeBVUA8sERVd4jIcyIyrITFZwJ1cf2y3wAsUNWtACKy\nCFgPdBKRgyLyoHuZae7babcC1wO/LsuGGWOMKR+iqiW38nEikgbsL+PiYcDxciynvFhdpWN1lY7V\nVTq+WhdcXm1XqGqJ5+6rRVBcDhGJU9Vir5s4weoqHaurdKyu0vHVuqByaqvRYz0ZY4wpmQWFMcaY\nYllQwGynC7gEq6t0rK7SsbpKx1frgkqorcZfozDGGFM8O6IwxhhTrGodFCUNjy4i94lImseQ5hM8\n3hsvIknur/E+VFeBx/xlFy9bkXW529wlIjtFZIeIvOcx37H9VUJdju0vEZnhse5dInLK4z0nf76K\nq8vJ/dVaRNaIyCb3WG9DPd6b6l4uUURu8YW6RCRSRM557K83KrmuK0RktbumtSIS4fFe+f58qWq1\n/AL8gd1AW6AWsAXoelGb+4B/FbFsI2CP+3tD9+uGTtflfi/Twf3VAdh0fl8ATXxkfxVZl9P766L2\nvwLm+8L+ulRdTu8vXOfaf+F+3RXY5/F6CxCEa8y43YC/D9QVCWx3cH99AIx3vx4EvF1RP1/V+YjC\n2+HRi3IL8LmqpqvqSeBzLv3sjMqsqyJ5U9dDwEz3PkFd43eB8/vrUnVVpNL+O44BFrlfO72/LlVX\nRfKmLgXquV/X5//GlBsOLFbVHFXdCyRziTHgKrmuiuRNXV2B1e7XazzeL/efr+ocFCUOj+52p/vQ\nbamInB/80NtlK7sugNriGjU3RkRuK6eavK2rI9BRRNa51z+4FMs6URc4u78A1ykCXH8Jf1naZSu5\nLnB2f/0RGCciB4HluI52vF3WiboA2rhPSX0lIteUU03e1rUFuNP9+nYgVEQae7lsqVTnoCh2eHS3\n/wKRqtoD+AJYWIplnagLoLW6emGOBV4RkXaVWFcArtM81+H6S3SuiDTwclkn6gJn99d5o4GlqlpQ\nhmVL63LqAmf31xjgTVWNAIYCb4vrMQZO769L1XUE1/6KAp4E3hORepQPb+p6CvipiGwCfgocAvK9\nXLZUqnNQlDQ8Oqp6QlVz3JNzcD2u1atlHaoLVT3s/r4HWAtEVVZd7jYfq2vY+L1AIq5f0I7ur2Lq\ncnp/nTeaH5/ecXp/Xaoup/fXg8AS9/rXA7VxjWPk9P4qsi73qbAT7vkbcV1T6FhZdanqYVW9wx1U\nz7rnZXi5TaVTERdifOEL11+Ze3AdWp+/GHTlRW2ae7y+HYjR/7sYtBfXhaCG7teNfKCuhrieQQ6u\n/0BJFHOhsgLqGgws9Fh/CtDYB/bXpepydH+523UC9uHus+QLP1/F1OX0z9cK4D736y64frkJcCU/\nvpi9h/K7mH05dYWfrwPXRedDlfxzHwb4uV+/ADxXUT9fl71BvvyF6zBxF66kf9Y97zlcz8sAeBHY\n4f5HWAN09lj2AVwXzZKB+32hLuAqYJt7/jbgwUquS4C/Azvd6x/tI/uryLqc3l/u6T8CLxWxrGP7\n61J1Ob2/cF2cXede/2bgZo9ln3UvlwgM8YW6cF0fOP//9Afg55Vc1whcYb4LmIs75Cvi58t6Zhtj\njClWdb5GYYwxphxYUBhjjCmWBYUxxphiWVAYY4wplgWFMcaYYllQGGOMKZYFhTHGmGJZUBhjjCnW\n/wcjds13E+AIqAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105345410>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再调小一点，由于特征包括原始特征+tfidf特征，是多了些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 2 candidates, totalling 6 fits\n",
      "[CV] colsample_bytree=0.3 ............................................\n",
      "[CV] colsample_bytree=0.3 ............................................\n",
      "[CV] colsample_bytree=0.3 ............................................\n",
      "[CV] colsample_bytree=0.4 ............................................\n",
      "[CV] ...... colsample_bytree=0.3, score=-0.488641754222, total= 1.4min\n",
      "[CV] colsample_bytree=0.4 ............................................\n",
      "[CV] ...... colsample_bytree=0.4, score=-0.471873895047, total= 1.4min\n",
      "[CV] ...... colsample_bytree=0.3, score=-0.473950284714, total= 1.4min\n",
      "[CV] ...... colsample_bytree=0.3, score=-0.482898581296, total= 1.5min\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   3 out of   6 | elapsed:  2.1min remaining:  2.1min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... colsample_bytree=0.4, score=-0.483356576857, total=  31.9s\n",
      "[CV] ...... colsample_bytree=0.4, score=-0.487104782066, total=  30.7s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   6 out of   6 | elapsed:  2.7min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', colsample_bytree=1.0, learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=10, min_child_weight=5,\n",
       "        min_split_gain=0.0, n_estimators=374, n_jobs=4, num_class=9,\n",
       "        num_leaves=70, objective='multiclass', random_state=0,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=50000, subsample_freq=1),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.3, 0.4]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':10,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(3,5)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.480777925702\n",
      "{'colsample_bytree': 0.4}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### colsample_bytree=0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)感觉不用调了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 减小学习率，调整n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('best n_estimators:', 2674)\n",
      "('best cv score:', 0.47501497420832611)\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':10,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用所有训练数据，采用最佳参数重新训练模型\n",
    "由于样本数目增多，模型复杂度稍微扩大一点？\n",
    "num_leaves增多5\n",
    "#min_child_samples按样本比例增加到15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='goss', colsample_bytree=0.4, learning_rate=0.01,\n",
       "        max_bin=127, max_depth=7, min_child_samples=15, min_child_weight=5,\n",
       "        min_split_gain=0.0, n_estimators=2674, n_jobs=4, num_class=9,\n",
       "        num_leaves=75, objective='multiclass', random_state=0,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=50000, subsample_freq=1)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'n_estimators':n_estimators_2,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':75,\n",
    "          'min_child_samples':15,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "lg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle.dump(lg, open(\"Otto_LightGBM_goss_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"columns\":list(feat_names), \"importance\":list(lg.feature_importances_.T)})\n",
    "df = df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>feat_67_tfidf</td>\n",
       "      <td>25146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>feat_25_tfidf</td>\n",
       "      <td>23815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>feat_24_tfidf</td>\n",
       "      <td>22679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>feat_48_tfidf</td>\n",
       "      <td>22164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>feat_40_tfidf</td>\n",
       "      <td>21094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>feat_86_tfidf</td>\n",
       "      <td>20300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>feat_14_tfidf</td>\n",
       "      <td>17736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>feat_62_tfidf</td>\n",
       "      <td>16519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>feat_16_tfidf</td>\n",
       "      <td>14709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>feat_64_tfidf</td>\n",
       "      <td>14528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>feat_88_tfidf</td>\n",
       "      <td>14076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>feat_33_tfidf</td>\n",
       "      <td>13858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>feat_15_tfidf</td>\n",
       "      <td>13388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>feat_67</td>\n",
       "      <td>13220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>feat_42_tfidf</td>\n",
       "      <td>12693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>feat_70_tfidf</td>\n",
       "      <td>12559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>feat_54_tfidf</td>\n",
       "      <td>12352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>feat_34_tfidf</td>\n",
       "      <td>11506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>feat_8_tfidf</td>\n",
       "      <td>11079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>feat_24</td>\n",
       "      <td>10924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>feat_32_tfidf</td>\n",
       "      <td>10902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>feat_72_tfidf</td>\n",
       "      <td>10635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>feat_43_tfidf</td>\n",
       "      <td>10540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>feat_25</td>\n",
       "      <td>10501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>feat_60_tfidf</td>\n",
       "      <td>10065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>feat_48</td>\n",
       "      <td>9659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>feat_86</td>\n",
       "      <td>9450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>feat_85_tfidf</td>\n",
       "      <td>9426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>feat_36_tfidf</td>\n",
       "      <td>9285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>feat_71_tfidf</td>\n",
       "      <td>9177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>feat_30</td>\n",
       "      <td>1195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>feat_3</td>\n",
       "      <td>1177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>feat_84_tfidf</td>\n",
       "      <td>1174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>feat_7</td>\n",
       "      <td>1162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>feat_45</td>\n",
       "      <td>1087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>feat_23</td>\n",
       "      <td>1079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>feat_27</td>\n",
       "      <td>1032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>feat_49</td>\n",
       "      <td>1021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>feat_82_tfidf</td>\n",
       "      <td>1003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>feat_63</td>\n",
       "      <td>996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>feat_19</td>\n",
       "      <td>892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>feat_6_tfidf</td>\n",
       "      <td>864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>feat_21</td>\n",
       "      <td>832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>feat_52</td>\n",
       "      <td>816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>feat_46</td>\n",
       "      <td>788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>feat_61_tfidf</td>\n",
       "      <td>774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>feat_77</td>\n",
       "      <td>748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>feat_2</td>\n",
       "      <td>720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>feat_51_tfidf</td>\n",
       "      <td>708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>feat_12</td>\n",
       "      <td>665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>feat_31</td>\n",
       "      <td>658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>feat_5</td>\n",
       "      <td>657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>feat_81</td>\n",
       "      <td>614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>feat_93</td>\n",
       "      <td>593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>feat_28</td>\n",
       "      <td>574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>feat_61</td>\n",
       "      <td>377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>feat_84</td>\n",
       "      <td>370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>feat_82</td>\n",
       "      <td>245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>feat_6</td>\n",
       "      <td>223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>feat_51</td>\n",
       "      <td>199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           columns  importance\n",
       "159  feat_67_tfidf       25146\n",
       "117  feat_25_tfidf       23815\n",
       "116  feat_24_tfidf       22679\n",
       "140  feat_48_tfidf       22164\n",
       "132  feat_40_tfidf       21094\n",
       "178  feat_86_tfidf       20300\n",
       "106  feat_14_tfidf       17736\n",
       "154  feat_62_tfidf       16519\n",
       "108  feat_16_tfidf       14709\n",
       "156  feat_64_tfidf       14528\n",
       "180  feat_88_tfidf       14076\n",
       "125  feat_33_tfidf       13858\n",
       "107  feat_15_tfidf       13388\n",
       "66         feat_67       13220\n",
       "134  feat_42_tfidf       12693\n",
       "162  feat_70_tfidf       12559\n",
       "146  feat_54_tfidf       12352\n",
       "126  feat_34_tfidf       11506\n",
       "100   feat_8_tfidf       11079\n",
       "23         feat_24       10924\n",
       "124  feat_32_tfidf       10902\n",
       "164  feat_72_tfidf       10635\n",
       "135  feat_43_tfidf       10540\n",
       "24         feat_25       10501\n",
       "152  feat_60_tfidf       10065\n",
       "47         feat_48        9659\n",
       "85         feat_86        9450\n",
       "177  feat_85_tfidf        9426\n",
       "128  feat_36_tfidf        9285\n",
       "163  feat_71_tfidf        9177\n",
       "..             ...         ...\n",
       "29         feat_30        1195\n",
       "2           feat_3        1177\n",
       "176  feat_84_tfidf        1174\n",
       "6           feat_7        1162\n",
       "44         feat_45        1087\n",
       "22         feat_23        1079\n",
       "26         feat_27        1032\n",
       "48         feat_49        1021\n",
       "174  feat_82_tfidf        1003\n",
       "62         feat_63         996\n",
       "18         feat_19         892\n",
       "98    feat_6_tfidf         864\n",
       "20         feat_21         832\n",
       "51         feat_52         816\n",
       "45         feat_46         788\n",
       "153  feat_61_tfidf         774\n",
       "76         feat_77         748\n",
       "1           feat_2         720\n",
       "143  feat_51_tfidf         708\n",
       "11         feat_12         665\n",
       "30         feat_31         658\n",
       "4           feat_5         657\n",
       "80         feat_81         614\n",
       "92         feat_93         593\n",
       "27         feat_28         574\n",
       "60         feat_61         377\n",
       "83         feat_84         370\n",
       "81         feat_82         245\n",
       "5           feat_6         223\n",
       "50         feat_51         199\n",
       "\n",
       "[186 rows x 2 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFIpJREFUeJzt3X/sZXV95/Hna0FMt+oCMhAC2EEz\nbUqbLNIJkrht3LqBAXc72K0byEYmLpvpNrDRbDdxrMlitCa4G21CYmkwToTGimzVMFnGxQkha5oU\nZEDkR5HOFKmMzMLoUCVxo4t97x/3891e5tzvj/n+Oufe+3wkN/fe9z3nft/n3HPP654f935TVUiS\nNO4f9d2AJGl4DAdJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOk7tu4HVOuuss2rr\n1q19tyFJU+Xhhx/+flVtWW64qQ2HrVu3cvDgwb7bkKSpkuRvVzKcu5UkSR2GgySpw3CQJHUYDpKk\nDsNBktRhOEiSOpYNhyQXJLk/yVNJnkzy/lb/SJLvJXm0Xa4aG+dDSQ4neTrJFWP1Ha12OMmesfqF\nSR5McijJF5Octt4TKklauZVsObwC/H5V/TJwGXBDkovaY39UVRe3y36A9tg1wK8AO4A/TnJKklOA\nTwNXAhcB1449zyfac20DXgKuX6fpkyStwrLhUFVHq+qRdvtl4CngvCVG2QncWVU/qarvAIeBS9vl\ncFU9U1U/Be4EdiYJ8JvAn7fxbweuXu0ESZLW7qSOOSTZCrwVeLCVbkzyWJK9Sc5otfOA58ZGO9Jq\ni9XfCPxdVb1yQl2SNtzWPfewdc89fbcxOCsOhySvA74EfKCqfgTcCrwFuBg4CnxyYdAJo9cq6pN6\n2J3kYJKDx44dW2nrkqSTtKJwSPIaRsHw+ar6MkBVvVBVP6uqvwc+w2i3EYw++V8wNvr5wPNL1L8P\nnJ7k1BPqHVV1W1Vtr6rtW7Ys+7tRkqRVWsnZSgE+CzxVVZ8aq587Nti7gSfa7X3ANUlem+RCYBvw\nDeAhYFs7M+k0Rget91VVAfcDv9PG3wXcvbbJkiStxUp+lfXtwHuBx5M82mp/wOhso4sZ7QJ6Fvhd\ngKp6MsldwF8xOtPphqr6GUCSG4F7gVOAvVX1ZHu+DwJ3JvlD4JuMwkiS1JNlw6Gq/oLJxwX2LzHO\nx4GPT6jvnzReVT3DP+yWkiT1zG9IS5I6DAdJUofhIEnqMBwkSR2GgySpw3CQ5ow/FaGVMBwkSR2G\ngySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcpDnhl990MgwHSVKH4SBJ6jAcJEkdhoOk\n3m3dc4/HRAbGcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQtK78MttsMBwk\nSR2GgySpw3CQJHUYDpKkjmXDIckFSe5P8lSSJ5O8v9XPTHIgyaF2fUarJ8ktSQ4neSzJJWPPtasN\nfyjJrrH6ryV5vI1zS5JsxMRKklZmJVsOrwC/X1W/DFwG3JDkImAPcF9VbQPua/cBrgS2tctu4FYY\nhQlwE/A24FLgpoVAacPsHhtvx9onTZK0WsuGQ1UdrapH2u2XgaeA84CdwO1tsNuBq9vtncAdNfIA\ncHqSc4ErgANVdbyqXgIOADvaY2+oqr+sqgLuGHsuSZppQz3196SOOSTZCrwVeBA4p6qOwihAgLPb\nYOcBz42NdqTVlqofmVCXJPVkxeGQ5HXAl4APVNWPlhp0Qq1WUZ/Uw+4kB5McPHbs2HItS5JWaUXh\nkOQ1jILh81X15VZ+oe0Sol2/2OpHgAvGRj8feH6Z+vkT6h1VdVtVba+q7Vu2bFlJ65KkVVjJ2UoB\nPgs8VVWfGntoH7BwxtEu4O6x+nXtrKXLgB+23U73ApcnOaMdiL4cuLc99nKSy9rfum7suSRJPTh1\nBcO8HXgv8HiSR1vtD4CbgbuSXA98F3hPe2w/cBVwGPgx8D6Aqjqe5GPAQ224j1bV8Xb794DPAT8H\nfLVdJEk9WTYcquovmHxcAOCdE4Yv4IZFnmsvsHdC/SDwq8v1IknaHH5DWpLUYThIkjoMB0lSh+Eg\nSeowHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQpsRQ/xG9\nZpPhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcJM00v1m+OoaDJKnD\ncJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR3LhkOSvUleTPLEWO0jSb6X5NF2uWrssQ8l\nOZzk6SRXjNV3tNrhJHvG6hcmeTDJoSRfTHLaek6gJOnkrWTL4XPAjgn1P6qqi9tlP0CSi4BrgF9p\n4/xxklOSnAJ8GrgSuAi4tg0L8In2XNuAl4Dr1zJBkqS1WzYcqurrwPEVPt9O4M6q+klVfQc4DFza\nLoer6pmq+ilwJ7AzSYDfBP68jX87cPVJToMkaZ2t5ZjDjUkea7udzmi184DnxoY50mqL1d8I/F1V\nvXJCfaIku5McTHLw2LFja2hdkrSU1YbDrcBbgIuBo8AnWz0Thq1V1CeqqtuqantVbd+yZcvJdSxJ\nWrFTVzNSVb2wcDvJZ4D/0e4eAS4YG/R84Pl2e1L9+8DpSU5tWw/jw0uSerKqLYck547dfTewcCbT\nPuCaJK9NciGwDfgG8BCwrZ2ZdBqjg9b7qqqA+4HfaePvAu5eTU/SPPFnqLXRlt1ySPIF4B3AWUmO\nADcB70hyMaNdQM8CvwtQVU8muQv4K+AV4Iaq+ll7nhuBe4FTgL1V9WT7Ex8E7kzyh8A3gc+u29RJ\n0piFUH325nf13MnwLRsOVXXthPKiK/Cq+jjw8Qn1/cD+CfVnGJ3NJEkaCL8hLUnqMBwkSR2GgyQN\nxJBONDAcJEkdhoOkZQ3pE602h+EgSeowHCRJHYaDJKljVb+tJKkf7vvXZnHLQYPnClHafIaDJKnD\ncJAkdRgOkqQOw0HShtm65x6PGU0pw0GSTmCoGQ6SpAkMB0lTY94/zW8mw0GS1GE4SJI6DAdJUofh\nIEnqMBwkSR2GgySpw3CQtCJ+MWy+GA6SpA7DQZLUYThIkjoMB6ln7sfXEBkOA+JKQkPjMjm/DAdJ\nUofhIEnLmMctKMNhYDyXXLPIZXr6LBsOSfYmeTHJE2O1M5McSHKoXZ/R6klyS5LDSR5LcsnYOLva\n8IeS7Bqr/1qSx9s4tyTJek+kJOnkrGTL4XPAjhNqe4D7qmobcF+7D3AlsK1ddgO3wihMgJuAtwGX\nAjctBEobZvfYeCf+LUnSJls2HKrq68DxE8o7gdvb7duBq8fqd9TIA8DpSc4FrgAOVNXxqnoJOADs\naI+9oar+sqoKuGPsuSRJPVntMYdzquooQLs+u9XPA54bG+5Iqy1VPzKhPlGS3UkOJjl47NixVbYu\nadZ5jGPt1vuA9KTjBbWK+kRVdVtVba+q7Vu2bFlli5Kk5aw2HF5ou4Ro1y+2+hHggrHhzgeeX6Z+\n/oS6JKlHqw2HfcDCGUe7gLvH6te1s5YuA37YdjvdC1ye5Ix2IPpy4N722MtJLmtnKV039lwaEDfT\npfly6nIDJPkC8A7grCRHGJ11dDNwV5Lrge8C72mD7weuAg4DPwbeB1BVx5N8DHioDffRqlo4yP17\njM6I+jngq+0iSerRsuFQVdcu8tA7JwxbwA2LPM9eYO+E+kHgV5frQ+rD1j338OzN7+q7Dc2goS9b\nfkNa0qq4q3G2GQ6StIh5DkDDYQ7N8wIvjfO9sDjDYZ34g3mSZonhIEnqMBwk6SRt9p6CPvZMGA7S\nAGzUm99dnVotw0GS1GE4SJI6DAdpSnmGnDaS4SBNOQNCG8FwkKSeDTHgl/3hPUnq2xBXnrPOLQdJ\nUofhIEnqMBykAXI3ivpmOKh3rgil4TEcJG2aIX8QGHJvfTAcNCi+QaeTr9vsMRwkrRtDYnYYDtIM\nWelPavjTG+tjlueh4SBJ6jAcJEkdhsM6m+XNTOlE7p6aXYbDBvANI2naGQ6SBscPV/0zHCRJHYbD\nlPMT1ubxNFHNE8NB0obYyIA0fDee/+xH0swwNNaPWw6SppqBsDEMhxWahgVwGnpcME299s15NTum\n6bVcUzgkeTbJ40keTXKw1c5MciDJoXZ9RqsnyS1JDid5LMklY8+zqw1/KMmutU2SJGmt1mPL4Z9X\n1cVVtb3d3wPcV1XbgPvafYArgW3tshu4FUZhAtwEvA24FLhpIVC08abpk4ykzbMRu5V2Are327cD\nV4/V76iRB4DTk5wLXAEcqKrjVfUScADYsQF9aQnrFRJ9hI2njmqWDGV5Xms4FPC1JA8n2d1q51TV\nUYB2fXarnwc8NzbukVZbrC4taghvnnk1r/N+KCvtzbLWU1nfXlXPJzkbOJDk20sMmwm1WqLefYJR\nAO0GeNOb3nSyvUqaIfO0ou7DmrYcqur5dv0i8BVGxwxeaLuLaNcvtsGPABeMjX4+8PwS9Ul/77aq\n2l5V27ds2bKW1qVBcoWnoVh1OCT5+SSvX7gNXA48AewDFs442gXc3W7vA65rZy1dBvyw7Xa6F7g8\nyRntQPTlraYVmrfNXalP8/JeW8tupXOAryRZeJ4/q6r/meQh4K4k1wPfBd7Tht8PXAUcBn4MvA+g\nqo4n+RjwUBvuo1V1fA19SVJvtu65h2dvflffbazZqsOhqp4B/umE+g+Ad06oF3DDIs+1F9i72l42\n0rx8SpDmyUaswGdtXeE3pCX1amgr1aH10xfDYUZ5HEIny+VlWPp+PQwHaRF9vzmlPhkOmklDW7EP\nrR9pOYaDJKnDcJCkDTDNv1cGhoMkaQLDQYPlGVfzx9d7OAwHaYNN2wpv2vrVxjAcVsFPtNPD12l+\n+FqvL8NB66KvN6YrBGljzGU4bNYKxRXX/FjNa+3ysbmc3ydnrf/sR2uwsLBO+gGw9f5hsKX+loZp\n2k+F1HSbyy0HTac+V3LTtoKdhn6noceVmqVpWWA4rIPlFoxZXHBWYl6nW0vzhI7pYDjMoNW+8eb9\nTTuP0z9v06uVMxx00tayQpn3ldG8T/9qTPM8m+YPHIbDJpvWBWWWzOprMKvTpX4YDjPOFYaWM82f\nbufNZr5Onso6pzZyIZvW55b0D9xyOEmunE7eZs+zjf57C8/vsqBZ5paDljRLK8D1/mLhwnNKs8gt\nB80MV9Trx+MQMhy0KeZxZTNv06vZYjhoLs1jWEknw3BopmFFMQ09rpd5mlZpiOb+gPRSK6EhrqD8\ndVVpug1xvTKJWw4DMLSAGlo/kjbf3G85bBZXqktz/miWbcRp1BvNcOjJNKwMV9LjNEzH0DjPNo/z\nevXmerfSiQvOyZ7B4j+fmQ6emTSdfM36NdfhMC18k2yc9Z63vlaaFYMJhyQ7kjyd5HCSPX33I0nz\nbBDhkOQU4NPAlcBFwLVJLuq3K220oe7uGWJP0mYbRDgAlwKHq+qZqvopcCews69mXDmszCzNp1ma\nFmk9DCUczgOeG7t/pNUkST1IVfXdA0neA1xRVf++3X8vcGlV/ccThtsN7G53fwl4eg1/9izg+2sY\nfzMMvceh9wf2uF7scX0MocdfqKotyw00lO85HAEuGLt/PvD8iQNV1W3AbevxB5McrKrt6/FcG2Xo\nPQ69P7DH9WKP62MaelwwlN1KDwHbklyY5DTgGmBfzz1J0twaxJZDVb2S5EbgXuAUYG9VPdlzW5I0\ntwYRDgBVtR/Yv4l/cl12T22wofc49P7AHteLPa6PaegRGMgBaUnSsAzlmIMkaUDmLhyG+DMdSS5I\ncn+Sp5I8meT9rf6RJN9L8mi7XNVzn88mebz1crDVzkxyIMmhdn1Gj/390ti8ejTJj5J8oO/5mGRv\nkheTPDFWmzjfMnJLWz4fS3JJjz3+tyTfbn18Jcnprb41yf8Zm59/0mOPi762ST7U5uPTSa7osccv\njvX3bJJHW72X+bhiVTU3F0YHu/8GeDNwGvAt4KIB9HUucEm7/Xrgrxn9jMhHgP/cd39jfT4LnHVC\n7b8Ce9rtPcAn+u5z7LX+38Av9D0fgd8ALgGeWG6+AVcBXwUCXAY82GOPlwOnttufGOtx6/hwPc/H\nia9te/98C3gtcGF735/SR48nPP5J4L/0OR9Xepm3LYdB/UzHgqo6WlWPtNsvA08xPd8Q3wnc3m7f\nDlzdYy/j3gn8TVX9bd+NVNXXgeMnlBebbzuBO2rkAeD0JOf20WNVfa2qXml3H2D0/aPeLDIfF7MT\nuLOqflJV3wEOM3r/b6ilekwS4N8AX9joPtbDvIXD4H+mI8lW4K3Ag610Y9us39vnLpumgK8lebh9\nWx3gnKo6CqOQA87urbtXu4ZXvwmHNB9h8fk21GX03zHaollwYZJvJvlfSX69r6aaSa/tEOfjrwMv\nVNWhsdqQ5uOrzFs4ZEJtMKdrJXkd8CXgA1X1I+BW4C3AxcBRRpukfXp7VV3C6Ndzb0jyGz33M1H7\nIuVvAf+9lYY2H5cyuGU0yYeBV4DPt9JR4E1V9VbgPwF/luQNPbW32Gs7uPkIXMurP7AMaT52zFs4\nrOhnOvqQ5DWMguHzVfVlgKp6oap+VlV/D3yGTdgsXkpVPd+uXwS+0vp5YWG3R7t+sb8O/78rgUeq\n6gUY3nxsFptvg1pGk+wC/iXwb6vtKG+7an7Qbj/MaH/+L/bR3xKv7dDm46nAbwNfXKgNaT5OMm/h\nMMif6Wj7Ij8LPFVVnxqrj+9rfjfwxInjbpYkP5/k9Qu3GR2sfILR/NvVBtsF3N1Ph6/yqk9oQ5qP\nYxabb/uA69pZS5cBP1zY/bTZkuwAPgj8VlX9eKy+JaP/wUKSNwPbgGd66nGx13YfcE2S1ya5kFGP\n39js/sb8C+DbVXVkoTCk+ThR30fEN/vC6GyQv2aU0h/uu5/W0z9jtMn7GPBou1wF/CnweKvvA87t\nscc3Mzr741vAkwvzDngjcB9wqF2f2fO8/MfAD4B/MlbrdT4yCqqjwP9l9In2+sXmG6PdIZ9uy+fj\nwPYeezzMaL/9wjL5J23Yf92WgW8BjwD/qsceF31tgQ+3+fg0cGVfPbb654D/cMKwvczHlV78hrQk\nqWPeditJklbAcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR3/D1qd8heE6ZwKAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x104f82d50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(lg.feature_importances_)), lg.feature_importances_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tfidf的特征重要性更高一些。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
